Boyfriend too close with ex

I need advice. I've been seeing a man for 6 months now. I knew he was very close w his ex but then found out the frequently go out for dinner together. The first time I found out I actually caught them at the restaurant and he never introduced us, waited until she left the restaurant before coming over to my table. I've now found out he has been going out to dinner w her but lied and said he was home all night. He recently had a surgery and I offered to take him. He declined the offer and later told me she was taking him to the surgery. It was the last straw for me and i left him. He never apologized and instead told me I was drama. I also found out all this time he never even told her about me. Am I wrong?