Update 12/5/18 @ 10:44am – All of the 40+ remote VPN sites affected by this outage are up and the Internet and IP Phone services have been restored. Our carrier stated that “Services were migrated to spare fibers on a local providers network, thus restoring services to a stable state.”
Several floors of the APC use the original APC phone system and I’m being told their carrier has not restored services yet. The Office of General Services has a placed a support call with this carrier.
Though, the IP phones and Internet are back up and functioning at the APC, we are still “re-routed” through the Comcast cable modem and will need to cut back over to our main Internet connection. As Wednesday is our regularly scheduled maintenance window from 4:30pm – 6:00pm; we plan to cut back over to our regular services at this time. This cut over will cause an outage for the APC but it should not last for long. We will send out an update once services are running on their normal paths again.
—————————————————————————————————————————————————————————————————–
Update 12/4/18 @ 9:13PM – Repair crews have ceased splicing efforts because of safety concerns, as wires from the power company began to spark. While in the manhole, repairs crews confirmed that there was extensive damage to the cable in 5 or 6 points, and Field Operations are searching for an alternate route to temporarily restore services until a permanent fix can be achieved. Field Operations are unable to access the manhole to make repairs until the area has been confirmed safe by the power company.
—————————————————————————————————————————————————————————————————–
Update 12/4/18 @ 8:36PM – The Outside Plant Engineer confirmed that repair crews were able to access the manhole and begin preparations for repairs. There is no estimated time of restoral at this moment because of multiple providers beginning repairs in the same location. The Outside Plant Engineer is engaging to discover when CenturyLink will be given priority access to begin repairs.
—————————————————————————————————————————————————————————————————–
Update 12/4/18 @ 7:28PM – The Transport NOC is confirming with the Outside Plant Engineer that access to the manhole has been granted.
—————————————————————————————————————————————————————————————————–
Update 12/4/18 @ 3:08PM – The local utility provider has advised that access to the manhole will be permitted at 00:00 GMT. The CenturyLink Outside Plant Engineer (OSP) reports that the fiber splicing repair team will be on site at that time and the replacement cable has already been delivered to the site. The next update will be relayed once the repair team access the manhole and is able to asses the damage and begin repair activities.
—————————————————————————————————————————————————————————————————–
Update 12/4/2018 @ 1:41PM – The local utility provider has advised that progress was made on the primary repairs during the overnight repair window, however not all repairs were completed. The local utility provider continues with repair activities today and will advise later in the day if access will be granted this evening for fiber repair activities to commence.
—————————————————————————————————————————————————————————————————–
Update 12/3/18 @ 6:58PM – Today we re-routed the VPN sites and currently brought up 22 of the 38. We have utilized a Comcast connection at the APC to restore Internet and Network access to the building. Telephony has not been restored as the lines are physically damaged outside of the building and currently not the kind that can be re-routed.
Please note, the Comcast connection is not as “business class” or robust as the usual connection. At peak usage it will most likely slow down, but it will be better than a complete outage. Once services are restored and verified, we will switch back to the normal configuration. We hope these measures are a help and we will continue to provide updates on this page as we have them.
And for a status on the locations still reporting as down, you can find that on our status page at: https://archoit.org/system-status/
—————————————————————————————————————————————————————————————————–
Update 12/3/18 @ 2:15PM – The local utility company has reported that repairs in the manhole continue at this time and the primary repairs to the power cables will commence tonight. Access is expected to be granted on December 4th once the power cables have been repaired and the area is deemed safe. Several providers have facilities in the manhole, and as such, access will be staggered for each provider. The ETA for each provider is not yet known. The next update will be relayed on December 4th or new information becomes available.
—————————————————————————————————————————————————————————————————–
Update 12/3/18 @ 1:45PM – The Transport NOC has requested an update from the utility company regarding the status of the repairs to the exposed power lines and the possibility of gaining access to the manhole today to begin repairs on the damaged fiber cable. Updates will be provided as they are obtained from the local utility company regarding access or if additional information has been obtained.
—————————————————————————————————————————————————————————————————–
Update 12/3/18 @ 9:15AM – Additional investigations determined the two issues were a power outage in addition to an event approximately 15000 ft from the second testing site. Field Operations advised access for additional isolation of the second fault would not be available until 14:00 GMT, and as such repair efforts were placed on hold. Upon dispatching the following morning, it was determined services were being impacted by a single fiber cut, and not the commercial power outage as previously thought. Protected services were impacted as both the working and protected paths traverse the same span. The damage was caused by a manhole fire, and the local utility company was working on repairs upon arrival of Field Operations. The local utility company advised the power lines would need to be repaired to confirm the area was safe before CenturyLink Repair teams were allowed access. The Transport NOC continued to follow up with the local utility company throughout the weekend for repair, and repair crews were on standby awaiting access to be granted so repair efforts could immediately commence.
—————————————————————————————————————————————————————————————————–
Update 12/2/18 @ 12:55PM – Field Operations reports that exposed power lines remain present at the damage site. As such, the local utility provider cannot provide access as this poses a safety risk. At this point, Field Operations and the repair crew continue to await site access. Updates will be provided as they are obtained from the local utility company regarding access or if additional information has been obtained.
—————————————————————————————————————————————————————————————————–
Update @ 11:06AM – CenturyLink was cleared to begin working on the issue. From the update: “The Local Exchange Carrier (LEC) was informed the technician has arrived at the job site and is working to restore your services.” Hopefully we should see some progress soon. Updates will be provided as they are obtained from the local utility company regarding access or if additional information has been obtained.
—————————————————————————————————————————————————————————————————–
Update 11/30/18 @ 9:27AM – Event Field Operations advise the local utility company continues repair efforts at this time. The local utility company advises all surrounding manholes must be verified, as it is suspected multiple manhole fires impacted services. The local utility company advises a progress meeting will commence to assess the status of repairs. Updates will be provided as they are obtained from the local utility company regarding access or if additional information has been obtained.
—————————————————————————————————————————————————————————————————–
Update @ 8:45PM – The Transport NOC has advised that access has still not been granted as the local utility company continues to work to repair the damage to the power lines to ensure the area is deemed safe to allow CenturyLink repairs to proceed. It has been confirmed that the local utility company will continue to work throughout the night. Updates will be provided as they are obtained from the local utility company regarding access or if additional information has been obtained.
—————————————————————————————————————————————————————————————————–
Update @ 2:37PM – The local utility company has advised that Field Operations will not be provided access to the manhole until they have de-energized the damaged power lines and repair the damage. Updates will be provided as they are obtained from the local utility company regarding access or if additional information has been obtained.
—————————————————————————————————————————————————————————————————–
Due to a fire in center city Philadelphia, communications are offline to the APC building at 222 North 17th St. This facility houses file and print resources for several other Catholic Social Services sites in Philadelphia such as Casa Del Carmen, and these resources will not be accessible until the outage is restored. Stay tuned for updates.
