Is it normal to miss out on your 20s

I am a woman with aspergers, anxiety and depression and I am in my mid-to late 20s. I've never had sex because I have very low sex drive. I am now worried it will seem like a red flag to anyone new I meet that at my age I am still a virgin.
For a guy I dated 2 years ago when I admitted being a virgin to him to he saw it as a red flag because I dated my ex for so long and never slept him. At first he was honored and said we'd take it slow but quickly he ended up saying that's not normal and that he refuses to live in "celibacy".
I've never been to a party. I have been invited but I didn't want to go. I'm sensitive to noise and I don't mind drinking nowadays but I don't want to get wasted, I am very prone to headaches and alcohol sometimes triggers depressive episodes. I have never had a group of friends. I am always always at home with my mother, every weekend.
I worry if i've missed out on the best years of my life...
I haven't had any helpful experiences to know myself better. I think I am bi but I have no proof. I am worried about eventually settling down with one of the men I know and always wonder if I would've been happier with a woman. I have had great times such as with my ex but I feel they do not count because we did not have sex, and maybe that makes us former best friends more than exes but I don't see a problem with that. We have reconnected and we reunited in person 2 months ago after not seeing each other in years because he happened to be around and wanted to say hi. That to me is not a sign of a failed connection with somebody.
Then I see other people my age they have families and such and have experienced a lot. I feel like a kid in comparison and I avoid social interaction because of it. When I get in my 30s I will be too old to experiment around. It will be time to seriously start planning for children if I want to be able to have any. It puts a stress on me every year that passes. I worry I will be a restless woman in my older years. Once more experienced and less anxious I might feel ready to live a less peaceful life but it will be too late, I will be trapped... Maybe I will always be a timid person on the other hand but I change what I want week to week so I cant ever say for sure. Sometimes the thought of finally settling down never having to worry about dating again is wonderful, the next day it terrifies me and I want to be free, I feel suffocated.