sorry for not typing the solution(it was too late night)
just
putting sol in some time.
-------------------------------------------
a(a^2 - 3b^2) = 2 ;
b(3a^2 - b^2) = 11;
now as i had said put a=rsinQ
b=rcosQ
we get
r3SinQ(1-4cos2Q)=2-------1
r 3CosQ(3-4cos2Q)=11----------2
divide eq 1 and 2
tanQ(1-4cos2Q)=2/11(3-4cos2Q)
now gather terms of cos2Q
we get
(tanQ-6/11)=4cos2Q(tanQ-2/11)
sec2Q(tanQ-6/11)=4(tanQ-2/11)
put sec2Q=1+tan2Q
now put tanQ=t
we get the equation
11t3-6t2-33t+2=0
now put it like this
11t3-22t2+16t2-32t-t+2=0
11t2(t-2)+16t(t-2)-(t-2)=0
so t=2 is a factor
that means tanQ=2
sinQ=2/sqrt(5) cosQ=1/sqrt(5)
using this we conclude that answer is 5