I've had this issue for years. Initially I assumed that setting the "Billing Contact" on the Client Profile page would do the job, but alas all invoices related emails are still sent to the primary contact.
The only solution I know of is to make the billing contact the primary contact and then add the actual customer as a secondary contact and configure their settings so they receive everything except billing emails.
The downside of this is that everything appears in the billing contact's name instead of the actual customer.