Cron unstable segmentation fault / heap corruption / 500 read timeout

[eugenevdm]

Ever since a few days ago our cron is failing miserably.

 

The errors are mostly segmentation fault - and nothing else.

 

One day we received this:

 

"ALERT - linked list corrupt on efree() - heap corruption detected (attacker 'REMOTE_ADDR not set', file 'unknown')"

 

I tried moving from PHP to GET as per the automation settings but I get this:

"500 read timeout"

 

We have been running our CRON wrapped in this:

<?php
ini_set('memory_limit', '1024M');
ini_set('max_execution_time','4800');
require('/var/www/vhosts/domain.com/httpdocs/support/admin/cron.php');
?>

 

I tried it with and without the wrapper. The situation is very urgent and I have logged a support ticket with WHMCS as well. They normally get back very quickly but not this time and it makes me worry.

 

Could a Linux boffin please shed some light?

[itomic]

Did you get a resolution to this? We're having the same problem.

 

This person solved it by downgrading php: http://www.webhostingtalk.com/showthread.php?t=889065

 

This person solved it by raising the memory limit for php in php.ini: http://www.overdrivedesign.biz/blog/whcms-segmentation-fault-with-pop-php/

 

The former is not really and option for us. The latter we've tried (even setting the memory limit to as high as 1Gb) but with no success.

 

Any ideas anyone?

[WHMCS JamesX]

Could be indicative of memory corruption; among other things. It could be the script itself, an installed PHP extension, hardware issue, etc.

 

What are the messages in your error_log surrounding the ALERT entry? You can also utilize strace to find where the seg-fault is happening.

 

My first thought is, possibly, APC is causing a seg-fault which is then caught by suhosin as an attack.

Edited December 15, 2012 by WHMCS JamesX

[itomic]

Thanks for your reply.

 

Fyi: the script is running on a Rackspace VPS service, if that makes any difference.

 

I'll have to get my tech guy(s) to answer your questions.

 

Thx.

[tqhosting]

have you added any new modules recently? we had this mysterious problem with cron after we implemented a custom server module.

 

I don't remember exactly, but I think the problem was that the module didn't implement a function that whmcs was looking for.

[itomic]

Thanks for the suggestion. All I can think of is that we've been experimenting with enabling auto-responder messages in WHMCS for when a new ticket is raised... but my other tech guys might know more.

[WHMCS Chris]

segfaults are typically caused by PHP modules that are corrupt. Unfortunately, they're very difficult to investigate. I would ultimately recommend you increase the servers global php.ini rather than relying on ini_set as well.

 

However, if you do need to isolate it - you would want to use strace (a linux command utility) to do so.

 

The command would be something like:

 

# strace -Ffvt -s 4096 -o cron.out php /path/to/cron.php

- from there you would read the output of cron.out, find the segfault, and begin reading up to the last instances it hit.

 

I'd also recommend looking at your PHP modules and see which may recent, or compiled using nonphp source. ie: from sourceforge instead of php.

[itomic]

Thanks Chris. We'll report back what we find. The problem seems intermittent at the moment. Some support emails arrived into WHMCS last weekend, even though we'd not done anything obvious that might have changed things.