How should i deal with friend overstepping?

Hello, this might be a bit long so bear with me. I'll start by saying I'm a 26 year old female with slight Asperger's (autism spectrum disorder, main symptoms are sensory, some social anxiety and being overwhelmed easily). I recently met a guy with what could be described as "classic autism"; he's smart about certain subjects and is capable of basic self care, but he can't live independently or be left alone for an extended period of time. I met him through my friend who has known his family for a long time. She suggested I start working for him, basically doing light housekeeping and looking after him, and I agreed to it because I really need a job. I met his mom who is a really nice woman, and she agreed to it as well.

Anyway, the problem is, he's starting to overstep boundaries just a bit. Maybe it's my fault for not setting clear boundaries, but it started out he was asking me to hang out every day, but now it's getting kind of inappropriate, like he's asking if he can lick/suck on my belly button. I told him I wasn't comfortable doing anything sexual and that I'm not really looking for a relationship (partially true, if I do start a relationship with someone I want it to not be too high maintenance). But he continues to ask, and he wants me to sleep in a tent with him (I can't sleep outside because of my sensory issues). I told him that he and his mom are welcome to meet me for lunch or dinner, but he doesn't just want to do that.

How should I let him down easy? I can't just drop him completely, someone did that to me and it hurts. I'm not a mean person or anything like that; I like my alone time but I also have some good friends I hang out with when none of us are busy, as well as spending time with my family. If I get the job working with him it'll only be like 1-2 days a week, I live 30 minutes away by car and I don't always have the energy to drive all that way, yet he asks me almost daily. Any advice is appreciated. Thank y'all.