Well magiclko (waise what is ur real name if u don't mind) there is some mistake in ur calculation.

First since self enegry comes becaose of interaction between similar charges so it can never be negative (u can't have -KQ²/R)

 

Lets for calculation of potential energy we define Q charge at centre as 1 -Q on inner shell as 2 and +Q on outer shell as 3

 

The total potential energy has five terms

Self energy of 2=KQ²/(2r)

Self energy of 3=KQ²/(2R)

Interaction of 1&2=-KQ²/r

Interaction of 1&3=KQ²/R

Interaction of 2&3=-KQ²/R

 

So total self energy =KQ²/2(1/r+1/R)

Total interaction enrgy =-KQ²/r

Total potential energy=(-KQ²/2)(1/r-1/R)