javooooo Posted February 23, 2013 Share Posted February 23, 2013 Hi guys, Over the last few days I've noticed that our cron isn't finishing. It reaches the backup generating stage, and then doesn't complete. I also noticed that the cPanel CPU usage hits 100% during this period. We run the cron externally via Cronless as we can't get the cron to operate properly internally; however this solution has been working for the last few months. Can anyone suggest what's causing the issue? Cheers 0 Quote Link to comment Share on other sites More sharing options...
othellotech Posted February 25, 2013 Share Posted February 25, 2013 You need a more powerful server (and better disk I/O) 0 Quote Link to comment Share on other sites More sharing options...
WHMCS Support Manager WHMCS John Posted March 1, 2013 WHMCS Support Manager Share Posted March 1, 2013 Hi, Do you encounter any problems when generating the backup manually (via Utilities > System > Database Status > Download Database Backup)? 0 Quote Link to comment Share on other sites More sharing options...
ezyweb Posted March 4, 2013 Share Posted March 4, 2013 (edited) Hi John, I've also noticed that since the 1st March, all automatic backups have stopped in one of my WHMCS installations, but they continue to work fine on another. Since both of them are working from the same cPanel server, and both installations are the same WHMCS version, I'm at a loss to explain why this has broken. I've just run a manual backup (via Utilities > System > Database Status > Download Database Backup). It has not completed, and cPanel has just sent an "Excessive resource usage" alert with a process that has not been automatically killed. Something's definitely changed. I'm just not sure what it is. Any ideas? Edited March 4, 2013 by ezyweb 0 Quote Link to comment Share on other sites More sharing options...
othellotech Posted March 4, 2013 Share Posted March 4, 2013 I've just run a manual backup (via Utilities > System > Database Status > Download Database Backup). It has not completed, and cPanel has just sent an "Excessive resource usage" alert with a process that has not been automatically killed. There's your clue Something's definitely changed. I'm just not sure what it is. Any ideas? Size of the DB outgrown what can be done before your process killer kills it - adjust the settings or move the WHMCS to a more powerful box with better disk i/o 0 Quote Link to comment Share on other sites More sharing options...
brianr Posted March 4, 2013 Share Posted March 4, 2013 ...and cPanel has just sent an "Excessive resource usage" alert with a process that has not been automatically killed. Emphasis added... I think you missed a word in the reading there... 0 Quote Link to comment Share on other sites More sharing options...
ezyweb Posted March 4, 2013 Share Posted March 4, 2013 The database is only 175MB in size and is running on a cPanel VM with 8GB RAM, 8 CPUs assigned to it and is only driving a handful of WordPress and Joomla sites plus two WHMCS installations. There are only 5 other small, low spec VMs running on the host. The cPanel VM in question is our corporate webserver and is running off an extremely robust Dell host with ample hardware including two Xeon quad core CPUs, 32GB RAM and 15K SAS HDDs, so it's not like I'm taxing the host. So while I appreciate your input Rob, the response is a little glib and provides no beneficial insight. I'm sure you'd agree that dealing with the backup of a 175MB database shouldn't be causing these issues. 0 Quote Link to comment Share on other sites More sharing options...
othellotech Posted March 4, 2013 Share Posted March 4, 2013 (edited) I've just run a manual backup (via Utilities > System > Database Status > Download Database Backup). It has not completed, I've just run the same thing on a DB which is 603Mb and it took 87 seconds, if yours isn't finishing, watching top, io etc while it runs will give you the insight, and it is highly likely to be a bottleneck in processing/disk-io or a php setting - the test I did took load up to 12.3 for a while Edited March 4, 2013 by othellotech typo 0 Quote Link to comment Share on other sites More sharing options...
ezyweb Posted March 8, 2013 Share Posted March 8, 2013 Hi guys, I've heard back from both John and Don from WHMCS Support about this matter. They've told me the following: Don: "As your database size is currently around 175MB, it is too big for the built in backup tool to be able to handle. Unfortunately being that big simply means that for a PHP script to loop through and save that data into temporary memory in order to generate a full backup is not possible - either the memory limits or max execution time limits on the server side of things will kick in way before that completes and terminate the script." I then asked how our friend Rob "othello" Golding was managing to achieve a full backup of a 603MB database, and received the following answer: John: "I'm not sure how Othello is managing that, you'd be best off asking him directly. In our experience it's not possible to backup databases at or above about 20MB with a standard PHP configuration." Obviously I was having no problems with the daily backups of my database until it reached about 170MB, which tells me I had my PHP tweaked pretty well for my environment, and that this seems to have nothing to do with disk I/O or even CPU. Having said that, I'd be keen to hear from you, Rob, as to how you're achieving PHP-scripted backups of a 603MB WHMCS database, when it's supposedly not possible to do. Any insight on PHP/System settings you can provide that might help us get more out of the inbuilt backup feature would be much appreciated. Kerry 0 Quote Link to comment Share on other sites More sharing options...
WHMCS Chris Posted March 8, 2013 Share Posted March 8, 2013 You need a more powerful server (and better disk I/O) I'm not entirely sure this is the actual solution. Granted, WHMCS, as should any billing system be isolated. However, have you tried running the cron with the various flags to isolate where it's having issues aside from the backups? http://docs.whmcs.com/Crons#Option_Flags Before you start looking for a new box, determine your actual I/O wait, with `iostat`. You should ultimately be under 1% here, anything above you're looking at potential hardware issues, so at that time you may want to look at swapping disks, etc. Do you see issues when you make tarballs of large files? Or SQL dumps of the WHMCS database manually, (`mysqldump $DBNAME > $DBNAME.sql` )? 0 Quote Link to comment Share on other sites More sharing options...
ezyweb Posted March 8, 2013 Share Posted March 8, 2013 (edited) Hi Chris, I can confirm that this particular VM and its host, absolutely scream under any other conditions. We've only found two things that cause these kinds of CPU spikes .. 1) the WHMCS daily cron and 2) the latest ownCloud software which we've been testing. Besides that, if this VM runs at anything much higher that a 1% load, it would be working hard. We can create and decompress big tarballs, do manual MySQL dumps and so on, all with no issue at all and at full speed and performance .. but try to run the WHCMS cron .. and that's another story. I'll take your advice and try running the option flags against the cron, but I don't think it will make much difference, especially if Don and John believe that 20Mb+ shuts the backup feature down. iostats display the following: 03/08/2013 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.61 0.18 1.22 0.21 0.00 96.77 No red flags there that I can see, especially with a 0.21% iowait. Kerry Edited March 8, 2013 by ezyweb 0 Quote Link to comment Share on other sites More sharing options...
ezyweb Posted March 8, 2013 Share Posted March 8, 2013 (edited) I should also add that I ran iostat while the manual backup was running, and got the following results: %user %nice %system %iowait %steal %idle 1.61 0.18 1.22 0.21 0.00 96.77 There is no increase in iowait even with the script running, but running top reveals CPU is almost constantly at 100%. Edited March 8, 2013 by ezyweb 0 Quote Link to comment Share on other sites More sharing options...
WHMCS Chris Posted March 8, 2013 Share Posted March 8, 2013 Agreed, that's a normal I/O load. As you have a ticket open, can you PM me the ticket? If not, can you you open another with: ATTN: Chris So it gets flagged to me? Aside from backups, the cron shouldn't be crippling your server. 0 Quote Link to comment Share on other sites More sharing options...
ezyweb Posted March 8, 2013 Share Posted March 8, 2013 (edited) The thing is, Chris, the backups don't "cripple" my server in any way. Site load speeds might slow down very slightly, but the VM otherwise keeps performing quite normally .. just with its CPU at or near 100% for the duration of the backup part of the cron run. Every other aspect of the daily cron runs and completes exactly as normal. It's only the backup that does not complete. Additionally, an hour later, my second WHMCS installation's cron runs with exactly the same CPU load behaviour, but it seems because the database is only about 60MB, the backup also completes without issue. Also, the physical host server isn't even breaking a sweat. With CentOS/top reporting 100% usage, the actual host reports the VM in question is always running at below 50% of its allocated CPU capacity. That's a part of the reason this behaviour is so hard to nail down and deal with. That's why I suspect this has something to do with the php script itself, and has little if anything to do with hardware as othello has suggested. PM on its way to you. Edited March 8, 2013 by ezyweb 0 Quote Link to comment Share on other sites More sharing options...
othellotech Posted March 8, 2013 Share Posted March 8, 2013 In our experience it's not possible to backup databases at or above about 20MB with a standard PHP configuration. It'll be because of the way php is setup and config'd for our WHMCS box Is your php compiled with –enable-memory-limit ? What have you set it to in the .htaccess ? 0 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.