I never saw the admin menu option: Utilities > Update WHMCS and was using a custom admin template. For that reason I couldn't set the "Update channel" or "temporary update path".
So, I set my admin template back to "blend" and now saw the Utilities > Update WHMCS menu item and was able to configure the "Update channel" and "temporary upgrade path".
Update check now works correctly.
After that I deleted my custom admin template, made a copy of the "blend" template, then set that as my admin template.