WHMCS has an option to support IDN domains but it's limited to domain check.
It doesn't work for domain registration. So basically one can say that WHMCS doesn't support IDN domains.
If WHMCS supports IDN domain lookup (it converts correctly to punnycode during domain check) why not pass the punnycode also to the domain registrar when registering?
Currently customers can add IDN domains to the cart, pay for them but then you have to manually register them at the registrar but customer can't manage them afterwards in WHMCS, so it's useless.
It's just a line of code in the registrar modules to convert to punnycode before submiting the domain (if IDN support is enabled in admin area), I don't think this is a hard task.
If domain is not IDN than the UTF-8 to punnycode will not have any side effects so it would be safe to convert all calls to registrar module to punnycode when IDN support is enabled.