((A+3B)x(3A-B))^2

= (3(AxA) - AxB + 9(BxA) - 3(BxB))^2

=  (0 - AxB - 9(AxB) - 0 )^2

= (-10(AxB))^2

= (-10(1)(2)(sqrt3/2))^2

= 300