Hurricane totally raped my datacenter (neocron.org)
Soo.. Boom!
First I got this email...okay pretty standard...
Quote:
Dear Customer, UPDATE: The LGA11 facility reports that commercial utility power is no longer available and the site has successfully transferred to generator. Winds and rains have picked up considerably in the area of the facility. Site engineers have completed a walk-through and all systems appear to be functioning as designed. No impact to customers has been observed. We urge all Internap customers in our data centers to engage their own emergency preparedness plans. We will continue to monitor the progress of the storm and the status of our facilities, and immediately communicate any important updates or necessary emergency actions to our customers. We advise all Internap customers to avoid making any unnecessary trips to the data center during this time.
No big deal I guess if everything is going okay...
Then I get this about an hour and a half later
Quote:
Hurricane Update: LGA11
Dear Customer, **EMERGENCY NOTIFICATION** Please be advised that Internap's LGA11 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost. We urge all Internap customers to take necessary remote action, if shutting down equipment in a more graceful fashion is possible before the outage occurs. The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now. Agile customers are advised to begin shutting down their servers immediately. For our Pro-managed customers, the customer support team will begin shutting down your servers gracefully at 12:30am EST to avoid damage to your equipment. For our cloud customers, we will also being shutting down the infrastructure at this time. Internap will continue to work hard to assess the situation and our recovery plans, and will communicate those plans as soon as possible. Thank you for your patience and understanding during this crisis.
tl;dr neocron.org dc is flooding and their diesel generators went pbbt. They are abandoning ship. We are offline until further notice. derp
