"We reserve the right to refuse service to anyone." That just about covers it IMO.
Suppose the tables were turned, and a group of Neo Nazi skinheads demanded that a Jewish caterer serve food at their next rally. Is the business owner obligated to honor that request?
Businesses can't turn down business from everyone, or they wouldn't last very long. If they want to turn away potential customers for whatever reason, that's their decision...and they can suffer the consequences of lost revenue.
HUD and the fair housing laws oppose your view. Also, your scenario is far-fetched in comparison to the hypothetical above. By definition, a Nazi would not patronize a Jewish caterer. But a Black family might go looking for an apartment, and there's a good chance the person to decide whether they get it will be White.