Unusual order activity

[bear]

2 hours ago, WHMCS Juan said:

Specifically, if you see that the IP address is from a hosting provider, then it is safe to assume that none of the traffic from the hosting provider's ASN is legitimate user traffic (unless the hosting provider is a customer of yours).

So, if you get an order for a dedicated server from someone selling VPS accounts (or even a VPN at some company), they should be rejected because they are connecting from a company IP address? Might not be applicable in all cases, I'd suggest.

[snake]

if the asn belongs to an isp, blocking them would potentially block existing and potential customers as well.

 

So far the custom client field has solved the issues.

[Remitur]

UPDATE: I just experienced another very strange activity; and, being very strange, I guess it may be related to this issue.

Description: Three customers, who have been inactive for years, insert three different "add funds" orders; every order is for just 10€, and every order has a "credit card" payment.
The three orders have been inserted the very same day, and about at the very same time (from 19:19 to 19:21)
All of the three orders has been posted from the very same USA IP ( 23.155.24.6 )

I checked with one of the customers, and he denies not only the insertion of this order, but also confirm have not logged in ...

Please, verify if any of you have experienced this same issue.

 

Edited July 19 by Remitur

[Remitur]

11 minutes ago, Remitur said:

UPDATE: I just experienced another very strange activity; and, being very strange, I guess it may be related to this issue.

Description: Three customers, who have been inactive for years, insert three different "add funds" orders; every order is for just 10€, and every order has a "credit card" payment.
The three orders have been inserted the very same day, and about at the very same time (from 19:19 to 19:21)
All of the three orders has been posted from the very same USA IP ( 23.155.24.6 )

I checked with one of the customers, and he denies not only the insertion of this order, but also confirm have not logged in ...

Please, verify if any of you have experienced this same issue.

 

Further details:
- the bad guy logged in as the three users
- he tried also a login for a fourth customer, but failed
- all the activity (3 log-in, 3 add funds order, 3 log out + 1 unsuccessfully login) lasted just 3 minutes: too much time for a script, too little time for a human 
- no previous activity that could justify some kind of brute force

Being so, I guess someone recovered somewhere a db of compromised passwords, selected those belonging to our site (four of them) and succeded for 3 of them, because the fourth customer in the meanwhile changed his password.

[CHG]

We have seen the same situation here, we noticed on cron summary, some customers that were inactive for about 5 years become active with no products at all.

I really dont know what its going on here, We are very concerned. 

WHMCS team, why you don't say anything...?

 

BTW custom fields doesnt fix at all, fake orders still there.

[bear]

12 hours ago, Remitur said:

- all the activity (3 log-in, 3 add funds order, 3 log out + 1 unsuccessfully login) lasted just 3 minutes: too much time for a script, too little time for a human 
- no previous activity that could justify some kind of brute force

Being so, I guess someone recovered somewhere a db of compromised passwords, selected those belonging to our site (four of them) and succeded for 3 of them, because the fourth customer in the meanwhile changed his password.

Not too long for a script. Scripts can be programmed to wait between tries/executions.
I'd find it odd that they found 4 people elsewhere that also happened to be customers, unless they can be shown to be part of the same community or something of that nature. 
If it stops with those I'd guess some weird coincidence. If you see more, I'd start to wonder it it was your own database somehow.

That IP belongs to Microtronix-esolutions, an IT solution provider. A possible connection if you, or these client accounts, had ever used that company?

Edited July 20 by bear

[Remitur]

1 hour ago, bear said:

If you see more, I'd start to wonder it it was your own database somehow.

No way: even if the WHMCS would be affected by a leak, passwords are not recoverable, but only the hash.., and the hash would be useless to log in.
I'm wondering i.e. about RockYou2024 or similar databases...
I just checked the email addresses used by violated users on https://haveibeenpwned.com/ and - no surprise - all four of them are reported as pwned.
 

So, two different ideas:

1 - somewhere, there's some kind of stolen password database (stolen in the past using some kind of malware in the client's PC) where you can find passwords in the format site, username, password ... and in that database there're also our four violated users

2 - or the users used the very same password for different sites; the password was used i.e. on Canva, and it was leaked in May 2019 together with a few million of others ... and the bad guy just tried "let's see if this password works also on this site"
 

The 2 is less probable than the 1 (we have would seen more unsuccessful login attempts, and not just one vs. four...)

[bear]

1 hour ago, Remitur said:

all four of them are reported as pwned

Then there's the problem. To find them as your clients is surprising, unless there's a commonality with hacked username or email and so on that led them to your site. Or, a long shot, it's a random guess they're on your site in WHMCS. Not impossible, but really implausible. 

[WHMCS John]

Hi all,

In response to this order and registration activity you've been reporting, the product team have prioritised adding new tools to help protect against fraudulent activities, spam, and abuse in our next release.

As immediate potential mitigation, we are providing a patch for v8.10.1 to add reCAPTCHA v3 capabilities:

 

[bear]

How will v3 of Recaptcha mitigate an existing customer's account being logged into with the password? At best it might prevent/inconvenience a scripted attempt, but it would require the admin enabling it for all logins, not just new ones. 

[bnb]

On 7/22/2024 at 9:01 PM, WHMCS John said:

 

Unfortunately, reCaptcha v3 didn’t solve anything. It is no better than v2 or invisible alternative.

The only fix that worked for us was the custom field. That still works for now. 

[WHMCS John]

10 hours ago, bear said:

How will v3 of Recaptcha mitigate an existing customer's account being logged into with the password? At best it might prevent/inconvenience a scripted attempt, but it would require the admin enabling it for all logins, not just new ones. 

I don't think a new captcha will prevent actors from logging into the client area with the valid password.

The best protection against something like that would probably be Two-Factor Authentication.

20 minutes ago, bnb said:

Unfortunately, reCaptcha v3 didn’t solve anything. It is no better than v2 or invisible alternative.

The only fix that worked for us was the custom field. That still works for now. 

reCAPTCHA v3 operates differently to v2 or Invisible and learns based on traffic on your site, so might take some time to bed-in and for you to find the best threshold to use. For info see the documentation: https://developers.google.com/recaptcha/docs/v3#interpreting_the_score

[slim]

@WHMCS John - Has anyone at WHMCS actually verified that the registrations that were occurring were solving captcha? I doubt it very much. My hunch is they have found a way to register accounts by bypassing captcha.

I did an experiment - I changed the captcha type in WHMCS from reCAPTCHA v2 -> Invisible reCAPTCHA - and the registrations kept on coming.

I doubt a scripted robot could adapt to that sort of change - at least not so quickly. 

Can you confirm that the registrations were, in fact, solving the captcha feature and not bypassing/sidestepping it?

Edited July 23 by slim

[Vander Host]

5 hours ago, slim said:

 

Can you confirm that the registrations were, in fact, solving the captcha feature and not bypassing/sidestepping it?

We tried all the CAPTCHAs and it didn't solve our problem. We added a custom field and it solved our problem.

I think the fact that WHMCS is releasing a new CAPTCHA makes them think the old CAPTCHA wasn't effective or buggy or bypassable. Unfortunately folk in tech support don't or can't always give straight security answers so we keep on guessing and speculating.

Edited July 24 by Vander Host

[slim]

This new hotfix has this in the notes:

If the reCAPTCHA v3 score is above the threshold, the form submission is blocked and an error is displayed "Recaptcha verification failed: Unknown error"

This message should at least be configurable via our UI - Perhaps something a LOT more customer-friendly would be suitable.  If a legit end user gets this on checkout, its game over for that order. 

 

[bear]

On 7/22/2024 at 4:01 PM, WHMCS John said:

As immediate potential mitigation, we are providing a patch for v8.10.1 to add reCAPTCHA v3 capabilities:

 

11 hours ago, WHMCS John said:

I don't think a new captcha will prevent actors from logging into the client area with the valid password

Umm, ok. 
Since it's been established they're likely using an account password, I don't see why this was even suggested, but that's just me I suppose.

[WHMCS John]

8 hours ago, slim said:

This new hotfix has this in the notes:

If the reCAPTCHA v3 score is above the threshold, the form submission is blocked and an error is displayed "Recaptcha verification failed: Unknown error"

This message should at least be configurable via our UI - Perhaps something a LOT more customer-friendly would be suitable.  If a legit end user gets this on checkout, its game over for that order. 

 

Thanks for the feedback. I'm sure we'll be able to iterate on this during the pre-release period.

 

13 hours ago, Vander Host said:

We tried all the CAPTCHAs and it didn't solve our problem. We added a custom field and it solved our problem.

I think the fact that WHMCS is releasing a new CAPTCHA makes them think the old CAPTCHA wasn't effective or buggy or bypassable. Unfortunately folk in tech support don't or can't always give straight security answers so we keep on guessing and speculating.

Unfortunately it isn't too difficult to find information on bypassing older reCAPTCHA or tools and services to solve the puzzles, a quick Google will find plenty of resources. There are even plugins for Selenium which advertise they can solve reCAPTCHAs automatically.

Ultimately we can't solve for that, but we are providing a range of tools which are aimed at mitigating the impacts of automated actors or services which may reach your site.

The best defence will always be a multi-layered one; comprising things like customising forms, Web Application Firewalls, visitor behaviour analysis/captcha and fraud checks.

 

 

[bnb]

5 hours ago, WHMCS John said:

Ultimately we can't solve for that, but we are providing a range of tools which are aimed at mitigating the impacts of automated actors or services which may reach your site.

The best defence will always be a multi-layered one; comprising things like customising forms, Web Application Firewalls, visitor behaviour analysis/captcha and fraud checks.

Did you consider an alternative to reCaptcha such as Cloudflare Turnstile?
Since reCaptcha is so widely used, I guess trying to bypass it is a great challenge for many spammers/hackers and alike.
It would be interesting to see how turnstile behaves with this problem.

[slim]

@WHMCS John - Can you answer my question regarding weather anyone at WHMCS has looked into and confirmed if these registrations are solving Captcha OR if they have found a way to submit registrations/orders WITHOUT solving captcha (i.e there is a bug somewhere in your code that allows them to place an order without doing the captcha)..

This is my hunch.

[WHMCS John]

10 hours ago, slim said:

@WHMCS John - Can you answer my question regarding weather anyone at WHMCS has looked into and confirmed if these registrations are solving Captcha OR if they have found a way to submit registrations/orders WITHOUT solving captcha (i.e there is a bug somewhere in your code that allows them to place an order without doing the captcha)..

This is my hunch.

From the logs we've analysed, including those shared above in this thread, we do not currently have reason to believe the form is being submitted without completing CAPTCHA atleast once in the session.

10 hours ago, bnb said:

Did you consider an alternative to reCaptcha such as Cloudflare Turnstile?
Since reCaptcha is so widely used, I guess trying to bypass it is a great challenge for many spammers/hackers and alike.
It would be interesting to see how turnstile behaves with this problem.

Yes we did. The current intention is to also add hCaptcha during the 8.11 pre-release period.

[wsa]

14 hours ago, WHMCS John said:

Yes we did. The current intention is to also add hCaptcha during the 8.11 pre-release period.

On version 8.1.1 you planning to add Cloudflare Turnstile?

[Ethan L]

I would like to add here that in general the fraud detection and protections are extremely poor in WHMCS and need substantial work. There are multiple workflows that allow bad actors to bypass checks, and do things such as test credit cards (using auth transactions) without hitting any retry limits.  Some of these have been brought up before: https://whmcs.community/topic/232237-maximum-credit-card-retry-limit/

This attack is just the latest example of the systems in place not being up to scratch. We've implemented our own additional fraud detection system using the ClientAdd and ClientAreaPage hooks to push people into extra checks and limit credit card retries etc but we shouldn't have to.

WHMCS desperately needs to take fraud protection seriously, it's not something to be outsourced, it's not going to solved with a captcha and the community deserves solutions and customisable tools to manage our risks and options on this front.

 

[snake]

yea its like the email verification check doesn;t actually verify anything, it just sits there telling you they didn;t verify it, but doesn;t actually stop them doing anything.

[WHMCS John]

18 hours ago, wsa said:

On version 8.1.1 you planning to add Cloudflare Turnstile?

This is not currently planned for v8.11, but hCaptcha is.

[WHMCS John]

On 7/26/2024 at 5:01 AM, Ethan L said:

There are multiple workflows that allow bad actors to bypass checks, and do things such as test credit cards (using auth transactions) without hitting any retry limits.  Some of these have been brought up before: https://whmcs.community/topic/232237-maximum-credit-card-retry-limit/

Rate limiting based on IP may be of some limited benefit in situations like this. However as the discussion in this thread does point to the actions originating from a range of addresses, it might not be a panacea.

I didn't locate a request to rate limit orders or credit card attempts, so I invite you to submit that so we can start tracking demand: https://requests.whmcs.com