Its good that you are comfortable with mahematics and chemistry atleast.
Once you are comfortable with mathematics means you should not find any problem in picking the concepts of physics and thus the numericals.
Every student has got some or other weak points. So Don't you worry.
Try to make your weak point the strongest and accept the challenge of learning physics with conceptual solidarity.
Start with building conepts in theory to apply them while solving numericals. For this matter go through by hard with NCERT txt books starting with +1 for physics and its solved problems.
First gain confidence in solving some common problems of mechanics, then electrostatics, manetism and modern physics etc.
Once finishing NCERT txt books try H .C. Verma, and solve as many problems as possible.
Do post your problems, if any on GOIIT forum.
I am sure you will be most comfortable with physics.
Good luck.