Jump to content
websavers

WHMCS Licensing server issues right now

Recommended Posts

@Kian you mean this isnt mysteriously forgotten about across the world until yesterday setups on users servers? 

Oh I am shocked, I imagine all users affected are collectively gasping at the thought of this being something whmcs knew about and can't/won't fix! 

Thanks for bringing this up Kian, funnily enough I was reading one of your blog posts last night (had a little check up on Leaps progress) but really it's obvious when it's whmcs fault because like petulant teens they go quiet until confronted and then blame all but themselves. At this point, I don't think they have any real developers employed or stupid things like this wouldn't happen repeatedly! 

Share this post


Link to post
Share on other sites
Posted (edited)

Alrighty so obviously it's not great that the licensing servers went down, though service outages are ultimately expected at some point or another; it happens to everyone.

There's plenty of options for how to handle paid software and WHMCS has chosen an option that requires that we use their licensing system to ensure the software works. We don't explicitly want to have to license the software purely to keep it alive: but that's what WHMCS requires of us. (As a comparison, think of GPL software, like most of the WordPress ecosystem. We pay for licenses to get updates, not to be able to continue using the software at all). And so, since this is WHMCS's requirement (and not one we specifically *want*), the bare minimum they could do for us is ensure that their licensing system works flawlessly. Clearly this incident brought into question two key issues:

  1. That their servers weren't able to handle the problem gracefully despite seemingly having multiple fallback licensing servers (though again, I'm not overly annoyed about this part), and
  2. That the software does not handle things in an acceptable manner when the licensing servers are not available

In my opinion that second issue is the true problem here.

As WHMCS staff have stated both in tickets and I believe here in this thread, WHMCS *is* programmed to keep a license cache and keep the software going in the event of licensing server failures. However the bug that created all of these issues is that the license cache only works if the domain for which you access WHMCS remains 100% consistent. So if you access WHMCS (or bots do) occasionally through an alternate URL, or the server's IP address, that triggers a clearing of the license cache, forcing WHMCS to check in with the licensing servers again.

In our case, it was IP address access that was clearing the license cache. In another user's case further back in this thread, it was slight variations on their domain -- seemingly a domain alias that didn't 301 to the primary TLD. This meant a simple fix to prevent the license cache clearing going forward: no longer allow access from the server's IP. However the reality is since this is WHMCS's software and WHMCS's licensing system and servers, it's up to WHMCS to ensure that this bug is fixed in the software and not something end-users have to deal with.

I have suggested that they simply add some code that runs prior to any licensing checks that ensures WHMCS redirects to the System URL configured in WHMCS, therefore making all license checks occur against the correct URL. WHMCS staff have indicated the following in response:

Quote

WHMCS will generate relative URLs based upon the WHMCS System URL setting, but doesn't currently for an automatic redirect to that URL. Thanks for providing this suggestion, I've made a note and have fed that suggestion to our product team for their consideration in future.

If you also believe this should be fixed in the WHMCS code and that it's not something we should be required to repair for WHMCS (even though we *can* do so, and have done so), please like this post and let them know in a support ticket that you'd like to have a pre-license-check redirect added to their codebase.

Edited by websavers

Share this post


Link to post
Share on other sites
17 minutes ago, websavers said:

please like this post and let them know in a support ticket that you'd like to have a pre-license-check redirect added to their codebase

I let them know that in 2012. I'm still waiting. In the meantime I had to solve the problem myself (as always). Good luck 🤞

Share this post


Link to post
Share on other sites
4 hours ago, Kian said:

I let them know that in 2012. I'm still waiting.

You should start a feature request. It would be marginally faster that that. 😉

Share this post


Link to post
Share on other sites
Posted (edited)

I think it's happening again! 

Edited by ADz83

Share this post


Link to post
Share on other sites
3 minutes ago, ADz83 said:

I think it's happening again! 

I seem to be able to connect to all but g.licensing.whmcs.com without any troubles.

Share this post


Link to post
Share on other sites
3 minutes ago, websavers said:

I seem to be able to connect to all but g.licensing.whmcs.com without any troubles.

Yeah I'm ok again now.  Think I just jumped the gun.  This debacle left a better taste.

Share this post


Link to post
Share on other sites

It genuinely wouldn't have shocked me! Quite a large company have told me if it locks them out again through failure to implement a working system then they're just going to bypass the licencing system and continue paying! 

Its absolutely awful customers feel this way, awful that whmcs don't even respect customers enough to hold their hands up and apologise 

Share this post


Link to post
Share on other sites

Hi all,

Thanks for your valuable thoughts and feedback.  Whenever a major incident occurrs, like most organisations, we perform a review to understand the causes and see what changes can be made in future.

We saw fewer than 1% of users affected, meaning the fallback systems in place were largely effective. However we appreciate the impact for those affected users was significant, I'd like to take this opportunity to reiterate our apologies. 85% of those users who were impacted, did have some an underlying configuration or use-case which caused the local license cache to be unavailable for fall-back.


Several approaches for mitigating such a situation in future were considered (in addition to addressing the root cause of the degraded licensing performance) to discuss some ideas in brief:

* Adding warnings to the UI if the license cache is expired or invalid, may result in false positives. That is part of the normal license validation process (particularly when reissuing licenses) so most of the time would require no action to rectify anyway.
* Adding a redirect to the System URL isn't a viable solution, as it would prevent access to the admin area after moving to a new location. Access via the admin-supplied URI is necessary in that situation to login and update the System URL value.
* Therefore our response will involve introducing extra leeway on the local license key when used from a different location, and providing messaging to admins if attempting to access from an invalid location.


Our aim is to include these measures in the next release cycle (ie. 8.4.0). In the meantime if you have any concerns about this, please don't hesitate to contact our support team: https://www.whmcs.com/support

Share this post


Link to post
Share on other sites
1 hour ago, WHMCS John said:

Several approaches for mitigating such a situation in future were considered (in addition to addressing the root cause of the degraded licensing performance) to discuss some ideas in brief:

* Adding warnings to the UI if the license cache is expired or invalid, may result in false positives. That is part of the normal license validation process (particularly when reissuing licenses) so most of the time would require no action to rectify anyway.
* Adding a redirect to the System URL isn't a viable solution, as it would prevent access to the admin area after moving to a new location. Access via the admin-supplied URI is necessary in that situation to login and update the System URL value.
* Therefore our response will involve introducing extra leeway on the local license key when used from a different location, and providing messaging to admins if attempting to access from an invalid location.

Hey John, this seems mostly reasonable to me, however I would say that in terms of the system URL redirect, many web apps do this, like WordPress. You simply add documentation indicating how to change the URL after a relocation. You could even supply a PHP CLI script to help make it happen. If this is reasonable for other web apps, I see no reason it's not reasonable for WHMCS.

As for your selected solution: would that alert show to all admins no matter which URL they've logged in with, and only after someone (but not necessarily that admin) has attempted to access WHMCS from a URL other than the system URL? The reason I'm asking is that whomever logs in using the wrong URL (domain/IP) may not be the one who needs to see the alert to correct it. Further the WHMCS instance may be accessed by someone not even logging in at all, which apparently clears that license cache as well.

Edited by websavers

Share this post


Link to post
Share on other sites

So what were these underlying issues that users had? It keeps getting skipped over? 

I don't think it's unreasonable to ask for examples as that what the initial claim 

Share this post


Link to post
Share on other sites
15 minutes ago, Bigol'tastynuggets said:

So what were these underlying issues that users had? It keeps getting skipped over? 

I don't think it's unreasonable to ask for examples as that what the initial claim 

They did address them in tickets to those users. The gist is that the license cache would normally work great to prevent WHMCS outages when their license servers are down, however the cache is erased any time a WHMCS instance is accessed using a different URL, which could include an alternate domain, or an IP address if your web server is configured to access the WHMCS instance by IP. (Note that it doesn't need to be an actual admin of your WHMCS install accessing it via the alternate domain or IP -- it could just be bots pinging the site using that incorrect URL that cause the license cache to be erased, and so the actual WHMCS admin would never know its happening).

And so John's list above is a list of possible solutions to prevent the URL being used to check in with the licensing server from changing, thus ensuring the integrity of the license cache.

Edited by websavers

Share this post


Link to post
Share on other sites

Indeed, and we'll be making sure that the local license cache is  not wiped immediately on attempt to use from different location, only on valid response from our license system. That should help bots or misconfigured systems from accidentally causing it.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use & Guidelines and understand your posts will initially be pre-moderated