I've posted this before and it's relevant here, so I hope this helps...
The key is to befriend people. If you become friends with your spectators, they'll be on your side - as long as your magic is solid, they'll want to see more. Then it becomes more of a hang out thing, and the atmosphere will be much friendlier and easier for you. Don't worry about screwing up or being rejected. Before you start, take a few deep breaths. Expel out that energy with each breath. Take heart from the successful performances you've had, trust your practice, trust yourself.
You will get the occasional idiot, but at the end of the day, he's occasional, and he's an idiot, so you know, he's not really worth losing sleep over. The best way to get confidence though is just to grit your teeth and do it. Performance experience helps so much in getting rid of nerves. Performing is addictive for me, after I do one that goes well, I want to do it more often, and performances give you confidence.