WHMCS CEO Matt Posted March 11, 2009 WHMCS CEO Share Posted March 11, 2009 (edited) ResellerClub are currently down due to an apparent hardware failure. This means all domain registration and maangement capabilities from WHMCS will not be functioning with the ResellerClub modules. 1830 GMT: We have been dealing with a major hardware failure that has necessitated replacement of hardware and restoration of data. This is a long and tedious process and is expected to finish around 12:00 AM GMT on March 12. We are doing everything we can to bring the system up as soon as possible and will keep you posted. Rest assured that all operation will resume normally as soon as we are online again. 1700 GMT: The downtime has been extended by another 2 hours. We regret the inconvenience caused. Rest assured, we are in contact with the DataCenter to get the issue resolved at the earliest. 1500 GMT: The issue has been identified and it is due to a hardware failure at the Datacenter. We are currently in communication with the DataCenter and ETA provided to resolve the issue is of 3hours. 1400 GMT: Due to certain technical issues at our upstream datacenter provider, the Orderbox control panel and associated services are currently undergoing an unscheduled downtime. Affected services include the Orderbox control panels, Supersite/Partnersite, Whois services and the API. While you and your customers would not be able to add new orders or manage existing ones, the orders themselves will not be affected. All domain names, hosting packages, digital certificates and other services will continue to function seamlessly. Our system admin team is relentlessly in contact with the datacenter to get to the root of this issue and restore normal functioning as soon as possible. While we unfortunately do not have a timeline for resolution, we will keep you informed about progress being made on a regular basis. We regret any inconvenience this must cause you, and implore your patience. See http://www.resellerclub.com/support/ for the latest status. Matt Edited March 11, 2009 by Matt 18:30 Update from ResellerClub 0 Quote Link to comment Share on other sites More sharing options...
WebzPro Posted March 11, 2009 Share Posted March 11, 2009 Peachy, glad I only have a couple orders in limbo right. Thank you for the info. 0 Quote Link to comment Share on other sites More sharing options...
netearth Posted March 11, 2009 Share Posted March 11, 2009 Its actually the whole LogicBoxes platform, so we are also down, and some 30 other registrars at the moment..... Role on the fix this evening! 0 Quote Link to comment Share on other sites More sharing options...
XN-Matt Posted March 11, 2009 Share Posted March 11, 2009 12am apparently. So much for redundancy. 0 Quote Link to comment Share on other sites More sharing options...
Funboy Posted March 11, 2009 Share Posted March 11, 2009 XN-Matt, I second that! Any advance on 12am anyone? Or is it all bets off We regret any inconvenience this must cause you, and implore your patience. Well we must respect the above as it could happen to any of us. 0 Quote Link to comment Share on other sites More sharing options...
XN-Matt Posted March 11, 2009 Share Posted March 11, 2009 True.. but when you run MISSION CRITICAL services such as domain reg - their systems should have backups, of backups, of backups. Feel sorry for people who have expired domains that can't be renewed and so on. Even a DR location that comes into action when their first is down. I doubt 12, it was meant to be back at 4:30pm GMT. 0 Quote Link to comment Share on other sites More sharing options...
netearth Posted March 11, 2009 Share Posted March 11, 2009 (edited) It first of all went down at 11am this morning, then was told 1300 by LB, then 1800 without any problems, 1830 came and an update to be 0000... Redundancy yeah, what is that word again? When we release our registrar platform later this year, I can already tell you we will have a DR site in the UK, with master in USA. Fingers crossed for midnight then At least it wasnt just LB that went POP today, the site offering Michael Jackson tickets for the O2 arena also went pop too!! :snigger: Edited March 11, 2009 by netearth 0 Quote Link to comment Share on other sites More sharing options...
Dominic Posted March 11, 2009 Share Posted March 11, 2009 Very glad we decided to switch to Enom last week! Have a few thousand with Directi though... 0 Quote Link to comment Share on other sites More sharing options...
XN-Matt Posted March 12, 2009 Share Posted March 12, 2009 Edit [11:50 PM GMT]: The data restoration process is taking longer than anticipated and the system is now expected to be up at around 06:00 AM GMT on March 12. 0 Quote Link to comment Share on other sites More sharing options...
XN-Matt Posted March 12, 2009 Share Posted March 12, 2009 At least it wasnt just LB that went POP today, the site offering Michael Jackson tickets for the O2 arena also went pop too!! :snigger: It can happen anywhere. I used to be involved with the company that used to do the ticketing from Glastonbury.. one year, it was hell as the load balancer just died under the pressure. Poor thing. 0 Quote Link to comment Share on other sites More sharing options...
netearth Posted March 12, 2009 Share Posted March 12, 2009 0020 GMT: The data restoration process is taking longer than anticipated and the system is now expected to be up at around 06:00 AM GMT on March 12 to ensure that the data is consistent . While we are dealing with this issue, we are already working out a plan to ensure that this does not happen again. We apologize for the inconvenience and thank you for your patience 0 Quote Link to comment Share on other sites More sharing options...
WebzPro Posted March 12, 2009 Share Posted March 12, 2009 Now i is 8 AM. Makes you wonder what type of system they are actually running as a back up. You would think something this critical would be using something way more advanced that wold just take over in this type of situation. 0 Quote Link to comment Share on other sites More sharing options...
XN-Matt Posted March 12, 2009 Share Posted March 12, 2009 (edited) 0020 GMT: Please note that we will take until 08:00 AM GMT, March 12, in order to complete the data restoration process currently underway. The recovery process is extremely slow as of now, and we’re working on possible alternatives that will speed things up. We are extremely sensitive to the inconvenience this downtime has posed to your business and are working relentlessly to resume services within the least possible time frame. The redundancy measures in place will ensure that all data is recovered. Resellerclub and Support teams are available to assist in any way during this time; do feel free to contact us. We will continue our updates and keep you posted on the situation at hand. Thank you for your patience and cooperation. Redundancy? lol. I think not. Edited March 12, 2009 by XN-Matt 0 Quote Link to comment Share on other sites More sharing options...
XN-Matt Posted March 12, 2009 Share Posted March 12, 2009 Joy. Looking forward to renewals tomorrow morning. 0 Quote Link to comment Share on other sites More sharing options...
WebzPro Posted March 12, 2009 Share Posted March 12, 2009 Just got that email also. Really well explained. Ya know sometimes stuff happens. 0 Quote Link to comment Share on other sites More sharing options...
othellotech Posted March 12, 2009 Share Posted March 12, 2009 Redundancy yeah, what is that word again? Its in the dictionary - right under "realistic pricing" - maybe directi stopped reading when they hit the word "profit" ... Unfortunately downtime, loss of service/data, and lack of control all come as standard with any SaaS system - its why we rejected the LogicBox offering for our registry setup. 0 Quote Link to comment Share on other sites More sharing options...
WebzPro Posted March 12, 2009 Share Posted March 12, 2009 Came up about 30 minutes ago. Just got done pushing through my orders. 0 Quote Link to comment Share on other sites More sharing options...
Gazza Posted March 12, 2009 Share Posted March 12, 2009 there going to be busy today, just sorted my renewals and orders. 0 Quote Link to comment Share on other sites More sharing options...
chickendippers Posted March 12, 2009 Share Posted March 12, 2009 Meh, eNom's "redundant" DNS was down for hours a few weeks back; no one's perfect. At least DirectI have had the decency to keep us updated, didn't get a peep out of eNom. 0 Quote Link to comment Share on other sites More sharing options...
XN-Matt Posted March 12, 2009 Share Posted March 12, 2009 Funny how things go.. Enom seems to be "non-responsive" now. lol 0 Quote Link to comment Share on other sites More sharing options...
othellotech Posted March 13, 2009 Share Posted March 13, 2009 Meh, eNom's "redundant" DNS was down for hours a few weeks back; no one's perfect. At least DirectI have had the decency to keep us updated, didn't get a peep out of eNom. Which always raises the question ... You're a *host* WTF are you doing using a "free" DNS service ? There is a huge difference between a *service* being down, and taking offline 27 ICANN Registrars entire business... 0 Quote Link to comment Share on other sites More sharing options...
chickendippers Posted March 13, 2009 Share Posted March 13, 2009 Redundancy; if our DNS cluster goes down then our main website will still be available 0 Quote Link to comment Share on other sites More sharing options...
Team ResellerClub Posted March 13, 2009 Share Posted March 13, 2009 Hi Guys, Firstly, we'd like to inform you that our systems are currently functional and apologize for all the inconvenience caused. As you might already know, this downtime was a result of an unexpected hardware failure on one of our storage systems. Despite the adequate redundancy and failover, this resulted in a data inconsistency that needed to be fixed. This is for the first time, in the past 8 years, that we have faced such an issue at our end and are currently taking all the required measures to ensure that this NEVER happens again. Furthermore, as a goodwill gesture, to compensate for the downtime, we will charge a ZERO Markup untill tomorrow on all new Adds, Renewals and Transfer-Ins of all gTLD Domain Names starting from now till 14th March 2009 (Saturday), at 8.30am GMT. Please refer to our blog for further details. (blog.resellerclub.com) I hope this can make up for all the trouble caused to you and your Customers. Moving ahead, for anyone else who was inconvenienced by the interruption of services and/or irretrievable data, all of us at ResellerClub are deeply sorry. We assure you that all the required actions are being taken to mitigate this type of situation from ever taking place again. Best Regards, Team ResellerClub ----------------------- Sales@ResellerClub.com http://ResellerClub.com http://Blog.ResellerClub.com ----------------------- 0 Quote Link to comment Share on other sites More sharing options...
Dominic Posted March 13, 2009 Share Posted March 13, 2009 taking offline 27 ICANN Registrars entire business... Didn't realise it was the entire thing 0 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.