Is it normal to feel triggered by sex

Ive had people ask me if ive experienced sexual trauma due to having reactions of fear and panic by being alone with a man when we are not in a public place, and going to great lengths to avoid sex because I feel it would be triggering, violent and like it would cause me to break down. Its a real problem in my life that holds me back in life. I have no memory of having gone through any physical abuse. The only negative experience I remember ive had was my first boyfriend putting his fingers in me and holding me down against my will when I said no but that is not what messed me up I think. We were in a relationship.
Do I have repressed memories? I feel like I am avoiding something which I do not want to know about or deal with and I want nothing that can trigger it but I want a normal life too and to be able to sleep with men I date and am attracted to.