Should i stay at my miserable job for the money?

I have currently be working for a contractor doing odd jobs a few times a week sawing apart wood, moving things and essentially just doing all the dirty work nobody else wants to do, but now i am beginning to hate it. Initially when i started it was just supposed to be a one time thing. I knew the guy through a mutual friend and he asked me if i could move some items for him on the first day and i did. At my first day (i didn't know he would hire me from then onward) things went pretty smoothly although he harshly scolded me for not arranging the tools in the exact way he wanted, even though i had never done construction work before, but i let it pass and moved on. So as time went on i did jobs here and there and i begin to get deeper into that world, even though i never wanted to be their to begin with. He is a really strange person often going from being happy and friendly to a complete asshole to me depending on his situation (on days when he is mad about something that has nothing to do with me he will still treat me rudely even when i do a good job. I feel like have to walk on eggshell whenever i am near this guy because no matter what i do, i have always done something "wrong". He has been doing the job for so long that there is an exact and specific way he wants everything done and it has gone from being something that was a little irritating because he would yell at me to absolutely infuriating. I remember one occasion when he yelled at me for nearly 5 minutes for getting 2 extra pieces of wood on top of the 8 he requested because i didn't see the 2 that he had already put down. He is really intimidating and what i don't like about the whole deal is this job has drained the quality of my life.

I had a pretty decent life before i took this job. I was a typical young guy who played xbox, hung out with my friends and saved and invested my money. (i have another job as well that doesn't pay as well but can offer more hours if i request it), but now i'm always on edge. My hair is getting thinner and i'm always so stressed about what I'll have to deal with the next time I'm on site, Always worried about the next time i'll go to work, all to collect a somewhat bigger paycheck than my other job (and he is always 1 month late with the check too). I don't know what to do, i want to leave this job, but i don't want to just outright quit since i know him through a mutual friend and don't want to burn bridges... what should i do? I have college starting in September and want to focus on that more.. what should i say to him to end things smoothly or as an excuse to leave the job? I want my life back, and i don't think my lungs and ears can handle all of the dry wall dust and loud machinery anymore... This job is about $18 and hour, but it is starting to not be worth it compared to the $12 an hour job that i like....