I suppose if you were a babysitter, and you weren't required to get a business license, didn't make enough money to have to file taxes, and didn't advertise, you could choose your clients based on any criteria you wanted.

But if you get a business license, incorporate so you can't be personally sued, and most importantly, want the ability to advertise, then you're in the public marketplace.