Weird issue. WHMCS sometimes hangs completely (happens most of the time with checking domain).

[-M-]

I have experienced this several times myself. Sometimes, at random, our WHMCS website will hang completely.

 

Most of the time this happens when ordering a domainname (during availability check of domain). And when it happens the loader animation keeps loading and nothing happens.

Everything related to the WHMCS stops working (keeps loading). Also I cannot access admin gui of WHMCS either. Nothing works on it anymore.

 

If we wait long enough, we will finally get a time-out error: 504 Gateway Time-out (nginx)

Even stopping and restarting nginx after this doesn't help. The only solution is to reboot the server... Which is clearly not a solution...

 

I am running on CentOS 6, Plesk 12.5 and have NGINX enabled (with PHP 5.6.23 FPM application served by NGINX).

I also enabled NGINX logs. I am no expert, but I notice some weird stuff.

 

nginx logs (not only related to this domain obviously):

2016/07/05 11:02:34 [info] 14428#0: *48 client closed connection while waiting for request, client: 141.212.122.145, server: xx.xx.xx.xx:443
2016/07/05 11:03:31 [info] 14428#0: *49 client 216.218.206.67 closed keepalive connection
2016/07/05 11:04:54 [info] 14428#0: *51 client sent invalid method while reading client request line, client: 169.229.3.91, server: , request: "N▒.."
2016/07/05 11:13:55 [info] 14428#0: *63 SSL_do_handshake() failed (SSL: error:1408A10B:SSL routines:SSL3_GET_CLIENT_HELLO:wrong version number) while SSL handshaking, client: 216.218.206.67, server: xx.xx.xx.xx:443
2016/07/05 11:14:48 [info] 14428#0: *64 SSL_do_handshake() failed (SSL: error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher) while SSL handshaking, client: 216.218.206.67, server: xx.xx.xx.xx:443
2016/07/05 11:21:43 [info] 14428#0: *86 client sent invalid method while reading client request line, client: 169.229.3.91, server: , request: ".5."ɥ▒▒▒W▒▒▒▒▒ߥ▒Q▒*.}[sM77▒▒kj8D /.<logl▒.▒▒▒-8▒.▒▒|..▒▒.S.▒▒"
2016/07/05 11:29:08 [info] 14428#0: *89 client closed connection while waiting for request, client: 141.212.122.209, server: xx.xx.xx.xx:443

 

The IP's in there are not related to us or our servers. The once which were related to us I replaced with 'xx'.

It's weird because the website shouldn't even be accessible for outsiders, because I created rules so everyone is redirected to our old site, until our new website is ready.

 

 

Also I checked the website logs (in Plesk), but the only errors I can find are these:

2016-07-05 11:32:04	Error	xx.xx.xx.xx	504	GET /order/cart.php?gid=2 HTTP/1.1	http://domain.com/order/cart.php?a=add&pid=33	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0	176 	nginx access
2016-07-05 11:32:04	Error	xx.xx.xx.xx		14428#0: *95 upstream timed out (110: Connection timed out) while reading response header from upstream				nginx error
2016-07-05 11:32:42	Error	xx.xx.xx.xx	504	GET /order/administration/ HTTP/1.1		Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0	176 	nginx access
2016-07-05 11:32:42	Error	xx.xx.xx.xx		14428#0: *97 upstream timed out (110: Connection timed out) while reading response header from upstream				nginx error
2016-07-05 11:34:54	Error	xx.xx.xx.xx	499	GET /order/administration/ HTTP/1.1		Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0	0 	nginx access
2016-07-05 11:34:56	Error	xx.xx.xx.xx	499	GET /order/administration/ HTTP/1.1		Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0	0 	nginx access
2016-07-05 11:35:05	Error	xx.xx.xx.xx	499	GET / HTTP/1.1		Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0	0 	nginx access
2016-07-05 11:35:59	Error	xx.xx.xx.xx	499	GET / HTTP/1.1		Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0	0 	nginx access
2016-07-05 11:36:00	Error	xx.xx.xx.xx	499	GET / HTTP/1.1		Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0	0 	nginx access
2016-07-05 11:36:30	Error	xx.xx.xx.xx	499	GET / HTTP/1.1		Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0	0 	nginx access

 

Anyone has an idea?

 

The domain suggestions tool is only enabled for 5 TLD's. So that's not it.

I already increase (performance) settings in the nginx.conf, but that also doesn't help.

 

Resource usage is low and nothing special. This is a new server with good speed, memory and on SSD's.

I also searched the forums, but I couldn't find anything related or similar...

 

Maybe I should just switch from nginx to FastCGI and see if that causes any problems like this...?

 

Thank you in advance.

 

Regards

[WHMCS Nate]

Hello,

 

Given that this really is a server specific issue, your best bet to finding a resolution is opening a ticket with our support department so they can investigate the details of your server.

 

Have a great day,

 

Nate C

[WHMCS John]

Hi,

Please review Utilities > Logs > Activity Log at the time of performing a domain availability lookup. Do you see any errors?

The solutions for common errors are published at: http://docs.whmcs.com/Order_Form_Issues

 

If not, please try selecting just 3 or 4 TLDs for the Setup > General Settings > Domains tab > Standard Whois > Configure > Bulk Check TLDs option.

The more TLDs are selected here the longer every domain availability check will take. With enough TLDs selected, the process will eventually take so long it hits your server's timeout limit.

[-M-]

Hi,

Please review Utilities > Logs > Activity Log at the time of performing a domain availability lookup. Do you see any errors?

The solutions for common errors are published at: http://docs.whmcs.com/Order_Form_Issues

 

If not, please try selecting just 3 or 4 TLDs for the Setup > General Settings > Domains tab > Standard Whois > Configure > Bulk Check TLDs option.

The more TLDs are selected here the longer every domain availability check will take. With enough TLDs selected, the process will eventually take so long it hits your server's timeout limit.

 

No errors whatsoever in the activity log. Only a few errors in regarding to syntax errors in send emails, BUT that's because I am working on those currently. Nothing out of the ordinary for the rest.

 

Also, as mentioned in my first post, I already lowered the amount of TLD's. It doesn't matter that much (how much TLD's I use). I currently have it set at 5 TLD's or so.

 

I run a few tests again; if I run the domain availability quickly and several times after each other, the system will completely hang. Nothing is possible. I only get the 'loader' image and cannot access the admin gui either anymore.

 

If I check the logs in the Plesk interface of the domainname I can see only this:

 

mod_fcgid: read data timeout in 45 seconds, referer: http://domain.com/order/cart.php?a=add&domain=register

Premature end of script headers: cart.php, referer: http://domain.com/order/cart.php?a=add&domain=register

 

I don't know if you can tell from the above, but I already switched back from nginx to fast-cgi. Though it will time-out, not the whole website will crash or not respond.

 

I created a simple video example here:

It says it all...

 

With running PHP as Fast-CGI the site will respond again after 45 seconds or so...

...when running the site with NGINX instead, the site will not come back at all. The only solution is to restart all services, or reboot the server.

 

I am a bit scared to put our new website public because of this. Obviously someone will not do the same as me, as shown in the movie, however I think the same will happen when several people lookup a domainname if it's a available or not.

 

Any ideas, John??

[bigsean]

I've experienced the exact same issue. The checker will eventually eat up memory on the server:

 

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

11105 username 20 0 234m 55m 8072 S 0.0 1.4 0:00.53 /usr/bin/php /home/username/public_html/whmcsinstalldir/domainchecker.php

11165 username 20 0 234m 55m 8064 S 0.0 1.4 0:00.52 /usr/bin/php /home/username/public_html/whmcsinstalldir/domainchecker.php

 

The above is a snippet of the processes, there are tons of them and the processes don't finish/end.

 

His video is quite clear on how to reproduce the error.

 

Thanks

[-M-]

I've experienced the exact same issue. The checker will eventually eat up memory on the server:

 

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

11105 username 20 0 234m 55m 8072 S 0.0 1.4 0:00.53 /usr/bin/php /home/username/public_html/whmcsinstalldir/domainchecker.php

11165 username 20 0 234m 55m 8064 S 0.0 1.4 0:00.52 /usr/bin/php /home/username/public_html/whmcsinstalldir/domainchecker.php

 

The above is a snippet of the processes, there are tons of them and the processes don't finish/end.

 

His video is quite clear on how to reproduce the error.

 

Thanks

 

I *think* it's caused when you have 'domain suggestions' turned on. Because if I don't enable/use it, it doesn't happen. But I am not sure though.

[bigsean]

I *think* it's caused when you have 'domain suggestions' turned on. Because if I don't enable/use it, it doesn't happen. But I am not sure though.

 

Are you referring to the Enom name spinner in General settings under the Domain tab? I am not using that.

 

 

Thanks

[-M-]

Are you referring to the Enom name spinner in General settings under the Domain tab? I am not using that. Thanks

 

I am not using Enom stuff. I meant the "Standard WHOIS" with a few TLD's.

If I am not mistaken, it's the function "Additional TLDs to Search"

 

But maybe it's my mistake, shouldn't have mentioned "domain suggestions". I forgot that I re-translated everything.