If its a girl he's hanging out with he is cheating on you with her. If its a guy, your dude is a closet homosexual. Guys dont turn down ass so they can hang out with friends all the time. Unless he's fucked in the head.
Sooo, lets recap,
(a) your boyfriend is cheating on you
(b) fucked in the head.
In case of (a), dump him. In case of (b) dump him. You get the drift.
Of course theres also (c), which is he doesnt have any testosterone and he isnt really a man at all. In which case, dump him.
Instead of taking the time to pull him off his Moms tit or his boyfriends ass, move on. Four years is alot of time, but a lifetime is much longer.