To make the calculation simple,we assume centre of circle to be origin.Let P=(acosp,bsinp) be the fixed point on the circumference and Q be the variable point (acosk,bsink)
then,eqn of tangent to the circle at Q is xcosk+ysink=a
PQ,which is the hypotenuse of the triangle comes out to be 2asin(p-k)/2.PR,the perpendicular distance of R from the tangent comes out as 2asin2(p-k)/2.this gives QR=2asin(p-k)/2cos(p-k)/2
area=2a2sin3(p-k)/2cos(p-k)/2
TO obtain max area,differentiate w.r.t. k,which gives tan(p-k)/2=rt3 or (p-k)/2=600
so,the max area is 3rt3a2/8