10th July 2023, 23:32 BST
We are seeing that we have lost power again and have reached out to the datacenter.
10th July 2023, 23:58 BST
The datacenter is experiencing a fire and is in emergency power off. We don't have further information at this time. We'll update this posting as we have more information.
11th July 2023, 00:29 BST
We've been informed that an electrical room experienced a fire, was put out by retardant, and the datacenter is in emergency power off status at the requirement of on-site fire fighters.
At the moment we're not sure when power will be switched back on or when our technician will be able to enter the building.
11th July 2023, 02:00 BST
The datacenter has now provided us with an update, stating that an isolated fire occurred in a UPS in an electrical room and the fire was detected and extinguished by the fire suppression system.
Here is the rest of the statement:
Upon arrival, the local fire department followed NEC guidelines and likely local laws and general best practices for firefighters. They cut the power to the building, which caused the earlier power issue.
Electricians are currently on-site, awaiting permission from the fire department to enter the building and perform repairs on the UPS. Once the UPS is repaired, the HVAC system will be powered on to subcool the facility. This process is estimated to take 3-4 hours. Subsequently, power will be restored to the data halls, bringing our network and servers back online.
The datacenter manager has provided an ETA of later today (morning of the 11th in the Eastern Time Zone) for the power to be restored to our racks.
11th July 2023, 14:10 BST
We have just received the following which was sent by the datacenter a few hours ago:
After reviewing the site, the fire Marshall is requiring that we extensively clean the UPS devices and rooms before they will allow us to re-energize the site.
We have a vendor at the site currently who will be performing that cleanup.
11th July 2023, 14:25 BST
Power remains off at our data center in Secaucus/EWR1 per the local fire marshal.
Current status update from DC Ops:
Our remediation vendor and our team has worked through the night to clean the UPS' at the request of the fire marshal.
They have made significant progress and we hope to have the cleaning completed by mid-day, at which time we will engage the fire marshal to review the site.
Following their review, we hope to get a sign off from them so that we can start the reenergizing process. The reenergizing process can take 4-5 hours, as we need to turn up the critical infrastructure prior to any servers.
11th July 2023, 17:30 BST
Current status update from DC Ops:
The EWR Secaucus data center remains powered down at this time per the fire marshal. We continue to clean and ready the site for final approval by the fire marshal in order to re-energize the facilities critical equipment. Site management, the fire marshal, and electrical contractors will be meeting at 2PM EDT in an attempt to receive approval from the fire marshal to re-energize the site. We do not foresee any issues that would result in not receiving such approval.
Re-energizing critical equipment will take 4-5 hours. After this process, we will be energizing customer circuits and powering on all customer equipment. We will provide updates as to when customers will be allowed in the facility once approved by the fire marshal.
11th July 2023, 17:30 BST
Current status update from DC Ops:
The EWR Secaucus data center remains powered down at this time per the fire marshal. We continue to clean and ready the site for final approval by the fire marshal in order to re-energize the facilities critical equipment. Site management, the fire marshal, and electrical contractors will be meeting at 2PM EDT in an attempt to receive approval from the fire marshal to re-energize the site. We do not foresee any issues that would result in not receiving such approval.
Re-energizing critical equipment will take 4-5 hours. After this process, we will be energizing customer circuits and powering on all customer equipment. We will provide updates as to when customers will be allowed in the facility once approved by the fire marshal.
11th July 2023, 18:00 BST
The datacenter has provided us an update stating that they "need to re-energize the critical infra at the site and get it cooled down prior to giving customer access", this will take 4 to 5 hours assuming the Fire Marshall gives the all clear.
Hopefully our team can enter the building around 6/7PM EDT (which is 23:00/00:00 BST).
11th July 2023, 20:00 BST
We have had an update, looks like things are delayed.
The EWR Secaucus data centre remains powered down at this time per the fire marshal. Site management, the fire marshal, and electrical contractors are currently meeting to review the cleaning process and seek approval from the fire marshal to re-energise the site.
11th July 2023, 21:20 BST
We have received an update from the datacenter and it's not looking good:
The EWR Secaucus data centre remains powered down at this time per the fire marshal.
We have just concluded a meeting with the fire marshal, electrical inspectors, and our on-site management. Significant progress has been made in the cleaning effort. However, after reviewing the situation with the fire marshal, additional areas need to be cleaned, and certain components of the fire system must be replaced. The fire marshal has scheduled a follow-up review at 9am EDT on Wednesday (14:00 BST). We are diligently working with our vendors to fulfil these new requirements and have arranged for additional cleaning personnel to meet the fire marshal's deadline.
In preparation for allowing clients back on-site, the fire marshal has mandated a comprehensive test of the fire/life safety systems. This test will be conducted after utility power has been restored and the fire system components have been replaced. We have coordinated with the necessary vendors to carry out this work tomorrow.
Assuming everything proceeds as planned, clients will be permitted to return to the site and power up their servers no earlier than late in the day on Wednesday.
12th July 2023, 16:30 BST
We've had some good news from the datacenter, hopefully all back online tonight!
Our site inspection this morning went well and we have been granted authorization to restore utility power to the site and are currently working on re-energizing utility power to the facility. Our onsite team is working with the fire marshal and electrical inspectors, ensuring electrical system safety as we prepare to bring utility power back to the site.
Once that is completed, we will work towards bringing up our critical infrastructure systems. This will take approximately 5 hours.
While we are working on that, we will also be working on our fire/life safety systems as we need to replace some smoke detectors and have a full inspection of the fire system prior to allowing customers to enter the facility.
We will be sending out hourly updates as we make progress on bringing the facility back online.
12th July 2023, 18:00 BST
Another update from the datacenter has been posted:
We have completed the full site inspection with the fire marshal and the electrical inspector and utility power has been restored to the site.
We are now working to restore critical systems and our onsite team has energized the primary electrical equipment that powers the site. Concurrently, we are beginning work to bring the mechanical plant online. Additional engineers from other facilities are on site this morning to expedite site turn up.
The ETA for bringing up the critical infrastructure systems is approximately 5 hours.
We are planning for a late afternoon/early evening time frame when clients will be able to come back on site.
12th July 2023, 18:15 BST
We are now hopefully ~5 hours away from our technician being allowed on-site.
Once we have access we can start getting everything back online, starting with our network and remote access.
From there we can begin to power on all our servers and restore services.
We expect this process to take 1-2 hours as we need to bring up systems slowly and confirm all is ok with each one.
12th July 2023, 18:50 BST
We have had an update from the datacenter:
Our onsite team has energized the primary electrical equipment that powers the site, enabling us to bring our mechanical plant online. We are currently cooling the facility.
As we monitor for stability, we are focused on bringing up our electrical systems. In starting this process, we have identified an issue with powering up our fire panel as well as power systems that were powered by UPS3. While this will cause us a delay, we are working with our vendors for remediation.
We are currently at 25% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 7 hours. We are still planning for an evening time frame when clients will be able to come back on site. We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted.
12th July 2023, 20:10 BST
We have had an update from the datacenter:
Our onsite team is currently bringing our UPS Systems online. We have our UPS vendor onsite assisting with this as we bring up UPS-4 and then UPS-R after that.
As these systems are brought online, we will concurrently work on bringing carriers up.
We are currently at 30% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 6 hours. We are still planning for an evening time frame when clients will be able to come back on site.
12th July 2023, 22:00 BST
We have had an update from the datacenter:
Our onsite team is currently bringing our UPS Systems online. We have our UPS vendor onsite assisting us with this. We have brought UPS-4 online and are currently charging it's associated battery system. While bringing UPS-R online we have run into a minor issue that we are currently investigating.
We are currently at 35% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 5 hours. We are still planning for an evening time frame when clients will be able to come back on site.
In process with our power system re-energizing, we have been working on our fire system as well. We are currently sourcing materials to bring our fire system fully online and do not have an ETA for completion. Because of this and fire marshal compliance we will only be allowed to have supervised escorted customer access when we finish bringing up the critical infrastructure systems. We are currently sourcing additional personnel to assist us with this escort policy.
12th July 2023, 23:20 BST
We have had an update from the datacenter:
Our onsite team is currently bringing our UPS Systems online. We have our UPS vendor onsite assisting us with this. We have brought UPS-4 online. We resolved our issues with UPS-R and have brought it online and are currently charging it's associated battery system. We will now begin work on bringing UPS-1 online. After bringing the UPS Systems online, we will then fully transfer load to our downstream power distribution, which will enable us to start powering up individual customer cabinets.
We are currently at 40% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 4 hours. We are still planning for an evening time frame when clients will be able to come back on site.
We are currently sourcing materials to bring our fire system fully online and do not have an ETA for completion. Because of this and fire marshal compliance we will only be allowed to have supervised escorted customer access when we finish bringing up the critical infrastructure systems.
13th July 2023, 01:00 BST
We have had an update from the datacenter, note we're not part of UPS-1:
Our onsite team is currently bringing our UPS Systems online. We now have UPS-4 and UPS-R online. While bringing up UPS-1, we ran into an issue and we are unable to bring it online. We have engaged with our vendor and are finding a workaround to deliver power downstream to customer cabinets.
Due to the issue with UPS-1, we will not be automatically powering up all customer cabinets. We will be reaching out to you to let you know if you are part of the group of customers on unprotected/non-UPS backed utility power so that you can make the decision whether to energize your cabinets at that time.
We are currently at 50% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 3 hours. The current time frame for when clients will be able to come back onsite is approximately 10:30PM EDT.
We are currently sourcing materials to bring our fire system fully online and do not have an ETA for completion. Because of this and fire marshal compliance we will only be allowed to have supervised escorted customer access when we finish bringing up the critical infrastructure systems. We will have additional personnel onsite to assist us with this escort policy.
13th July 2023, 01:15 BST
The area our equipment is located in has now been energised, we're seeing systems coming back online.
13th July 2023, 01:35 BST
Remote access has been established and we have been running checks on our equipment, services are mostly restored.
13th July 2023, 03:20 BST
Full service has now been restored, if you are having any issues please submit a ticket to our team.
We apologise for the downtime in NY and thank you for your patience and understanding.