let them be tan a and tan b

so that tan a = ntanb

 

tan(a-b) = [tana - tanb]/ [1+tanatanb]

divide by tanb ,

put tana/tanb = n

 

so tan(a-b) = (n-1)/ [ {1/(tanb)} + ntanb ]

 

now, numerator is ok, we have to deal with the denominator

 

let tanb = r

so the denominator is (1/r) + nr

use AM >= GM

[(1/r) + nr ] / 2 >= n^1/2

so [ 1/r + nr ] >= 2 n^1/2

 

since we have to find the maximum value of tan²( a-b)

tan(a-b) should be maximum

so its denominator should be minimum

 

and we have already found the minimum value of the denominator as 2 n^1/2

 

so tan(a-b) 's maximum value will be [ n-1 ] / [ 2n^1/2 ]

 

and so the maximum value of tan²( a-b) will be

(n-1)²/ 4n