madam, may I just say..... any person that has to push another person down, has major issues.... Truth is and as much as it hurts, you will need to get rid of this guy, he is a negative distraction....

To be honest he does sound insecure and very manipulative, he has a hold on you

However I have learnt when people give advice (like myself and others) it always falls on deaf ears until something dramatic takes place...

Ok you obviously forgave him for some personal reason, but be real to your self tell him he has issues and check his response