Here is the solution (thanks to Nathan at WHMCS Support)
The "Invalid System Template Name" error typically occurs when a custom template directory name contains illegal characters. Rename the custom sub-directory in /templates so it meets the requirements specified in the docs: http://docs.whmcs.com/Client_Area_Template_Files#Creating_a_Custom_Template
Then re-select your Setup > General Settings > General Tab > Default Template setting and click Save Changes.