WHMCS Licensing server issues right now

[Bigol'tastynuggets]

@Kian you mean this isnt mysteriously forgotten about across the world until yesterday setups on users servers? 

Oh I am shocked, I imagine all users affected are collectively gasping at the thought of this being something whmcs knew about and can't/won't fix! 

Thanks for bringing this up Kian, funnily enough I was reading one of your blog posts last night (had a little check up on Leaps progress) but really it's obvious when it's whmcs fault because like petulant teens they go quiet until confronted and then blame all but themselves. At this point, I don't think they have any real developers employed or stupid things like this wouldn't happen repeatedly! 

[websavers]

Alrighty so obviously it's not great that the licensing servers went down, though service outages are ultimately expected at some point or another; it happens to everyone.

There's plenty of options for how to handle paid software and WHMCS has chosen an option that requires that we use their licensing system to ensure the software works. We don't explicitly want to have to license the software purely to keep it alive: but that's what WHMCS requires of us. (As a comparison, think of GPL software, like most of the WordPress ecosystem. We pay for licenses to get updates, not to be able to continue using the software at all). And so, since this is WHMCS's requirement (and not one we specifically *want*), the bare minimum they could do for us is ensure that their licensing system works flawlessly. Clearly this incident brought into question two key issues:

1. That their servers weren't able to handle the problem gracefully despite seemingly having multiple fallback licensing servers (though again, I'm not overly annoyed about this part), and
2. That the software does not handle things in an acceptable manner when the licensing servers are not available

In my opinion that second issue is the true problem here.

As WHMCS staff have stated both in tickets and I believe here in this thread, WHMCS *is* programmed to keep a license cache and keep the software going in the event of licensing server failures. However the bug that created all of these issues is that the license cache only works if the domain for which you access WHMCS remains 100% consistent. So if you access WHMCS (or bots do) occasionally through an alternate URL, or the server's IP address, that triggers a clearing of the license cache, forcing WHMCS to check in with the licensing servers again.

In our case, it was IP address access that was clearing the license cache. In another user's case further back in this thread, it was slight variations on their domain -- seemingly a domain alias that didn't 301 to the primary TLD. This meant a simple fix to prevent the license cache clearing going forward: no longer allow access from the server's IP. However the reality is since this is WHMCS's software and WHMCS's licensing system and servers, it's up to WHMCS to ensure that this bug is fixed in the software and not something end-users have to deal with.

I have suggested that they simply add some code that runs prior to any licensing checks that ensures WHMCS redirects to the System URL configured in WHMCS, therefore making all license checks occur against the correct URL. WHMCS staff have indicated the following in response:

Quote

WHMCS will generate relative URLs based upon the WHMCS System URL setting, but doesn't currently for an automatic redirect to that URL. Thanks for providing this suggestion, I've made a note and have fed that suggestion to our product team for their consideration in future.

If you also believe this should be fixed in the WHMCS code and that it's not something we should be required to repair for WHMCS (even though we *can* do so, and have done so), please like this post and let them know in a support ticket that you'd like to have a pre-license-check redirect added to their codebase.

Edited October 5, 2021 by websavers

[Kian]

17 minutes ago, websavers said:

please like this post and let them know in a support ticket that you'd like to have a pre-license-check redirect added to their codebase

I let them know that in 2012. I'm still waiting. In the meantime I had to solve the problem myself (as always). Good luck 🤞

[bear]

4 hours ago, Kian said:

I let them know that in 2012. I'm still waiting.

You should start a feature request. It would be marginally faster that that. 😉

[Bigol'tastynuggets]

How can they fit that in? We were all asking for market connect! 🤣

[ADz83]

I think it's happening again! 

Edited October 6, 2021 by ADz83

[websavers]

3 minutes ago, ADz83 said:

I think it's happening again! 

I seem to be able to connect to all but g.licensing.whmcs.com without any troubles.

[ADz83]

3 minutes ago, websavers said:

I seem to be able to connect to all but g.licensing.whmcs.com without any troubles.

Yeah I'm ok again now.  Think I just jumped the gun.  This debacle left a better taste.

[Bigol'tastynuggets]

It genuinely wouldn't have shocked me! Quite a large company have told me if it locks them out again through failure to implement a working system then they're just going to bypass the licencing system and continue paying! 

Its absolutely awful customers feel this way, awful that whmcs don't even respect customers enough to hold their hands up and apologise 

[WHMCS John]

Hi all,

Thanks for your valuable thoughts and feedback.  Whenever a major incident occurrs, like most organisations, we perform a review to understand the causes and see what changes can be made in future.

We saw fewer than 1% of users affected, meaning the fallback systems in place were largely effective. However we appreciate the impact for those affected users was significant, I'd like to take this opportunity to reiterate our apologies. 85% of those users who were impacted, did have some an underlying configuration or use-case which caused the local license cache to be unavailable for fall-back.

Several approaches for mitigating such a situation in future were considered (in addition to addressing the root cause of the degraded licensing performance) to discuss some ideas in brief:

* Adding warnings to the UI if the license cache is expired or invalid, may result in false positives. That is part of the normal license validation process (particularly when reissuing licenses) so most of the time would require no action to rectify anyway.
* Adding a redirect to the System URL isn't a viable solution, as it would prevent access to the admin area after moving to a new location. Access via the admin-supplied URI is necessary in that situation to login and update the System URL value.
* Therefore our response will involve introducing extra leeway on the local license key when used from a different location, and providing messaging to admins if attempting to access from an invalid location.

Our aim is to include these measures in the next release cycle (ie. 8.4.0). In the meantime if you have any concerns about this, please don't hesitate to contact our support team: https://www.whmcs.com/support

[websavers]

1 hour ago, WHMCS John said:

Several approaches for mitigating such a situation in future were considered (in addition to addressing the root cause of the degraded licensing performance) to discuss some ideas in brief:

* Adding warnings to the UI if the license cache is expired or invalid, may result in false positives. That is part of the normal license validation process (particularly when reissuing licenses) so most of the time would require no action to rectify anyway.
* Adding a redirect to the System URL isn't a viable solution, as it would prevent access to the admin area after moving to a new location. Access via the admin-supplied URI is necessary in that situation to login and update the System URL value.
* Therefore our response will involve introducing extra leeway on the local license key when used from a different location, and providing messaging to admins if attempting to access from an invalid location.

Hey John, this seems mostly reasonable to me, however I would say that in terms of the system URL redirect, many web apps do this, like WordPress. You simply add documentation indicating how to change the URL after a relocation. You could even supply a PHP CLI script to help make it happen. If this is reasonable for other web apps, I see no reason it's not reasonable for WHMCS.

As for your selected solution: would that alert show to all admins no matter which URL they've logged in with, and only after someone (but not necessarily that admin) has attempted to access WHMCS from a URL other than the system URL? The reason I'm asking is that whomever logs in using the wrong URL (domain/IP) may not be the one who needs to see the alert to correct it. Further the WHMCS instance may be accessed by someone not even logging in at all, which apparently clears that license cache as well.

Edited October 14, 2021 by websavers

[Bigol'tastynuggets]

So what were these underlying issues that users had? It keeps getting skipped over? 

I don't think it's unreasonable to ask for examples as that what the initial claim 

[websavers]

15 minutes ago, Bigol'tastynuggets said:

So what were these underlying issues that users had? It keeps getting skipped over? 

I don't think it's unreasonable to ask for examples as that what the initial claim 

They did address them in tickets to those users. The gist is that the license cache would normally work great to prevent WHMCS outages when their license servers are down, however the cache is erased any time a WHMCS instance is accessed using a different URL, which could include an alternate domain, or an IP address if your web server is configured to access the WHMCS instance by IP. (Note that it doesn't need to be an actual admin of your WHMCS install accessing it via the alternate domain or IP -- it could just be bots pinging the site using that incorrect URL that cause the license cache to be erased, and so the actual WHMCS admin would never know its happening).

And so John's list above is a list of possible solutions to prevent the URL being used to check in with the licensing server from changing, thus ensuring the integrity of the license cache.

Edited October 14, 2021 by websavers

[WHMCS John]

Indeed, and we'll be making sure that the local license cache is  not wiped immediately on attempt to use from different location, only on valid response from our license system. That should help bots or misconfigured systems from accidentally causing it.

[websavers]

I'd also like to add one additional request on this topic: that the WHMCS Client Area front-end *not* be brought down should there be licensing server problems. In other words, I think it's more than reasonable to only do license checks when accessing the WHMCS admin area. This way even if we're experiencing problems with WHMCS, it doesn't affect customers -- only admins.