To avoid my co-workers outside of work?

I’m not a huge fan of interacting with my co-workers outside of work. If I see them at the store or walking down the street I will more than likely go the opposite direction. I’m not even sure why I don’t want to interact with them. I guess I just don’t really want them to get to know me.

I already have tons of friends outside of my work life, and they’re people I’ve known since forever unlike these temporary people at my job.
I have been invited quite a bit to hangout with them but I’ve always politely decline. I’ve noticed though some have started to dislike me at work because I’m always making excuses not to go to parties or whatever. I don’t hate them. I think some of them are even great people, I just don’t see the point of knowing each other outside of the job.

I’ve always felt it was weird when people are super chill with people that they probably have had fights with, or talked crap about behind their backs. I don’t really feel like co-workers necessarily make the best friends. You’ll end up just hating each other eventually, after all work is nothing but drama.

Does anyone else understand where I’m coming from?