Please correct me if I am wrong, but I have a feeling that such problems have very little in common with a typical JEE qn.

 

I mean, they are looking for prospective engineers not number theorists.

 

Concentrate more on Calculus, Determinants, Coordinate Geometry, Vectors and Complex Numbers. A lot of maths that you learn in your 1st two years at IIT have to do with Calculus and Complex Numbers. Physics too begins with lot of Vector stuff. Coordinate Geometry is of enormous use to engineers. And the teachers in the Math departments teach and work in such areas. So these are the topics to look out for.