RESOLVED – System Wide Outage and Unplanned Maintenance – But More Work to Be Done

*** Update – 2:56AM 12/19/2024 ***
The outage is resolved. The provider finished their maintenance a little early. We still need to resolve the failover issue – hopefully over the next few days. But for now, systems are normal!
 
There is/was a known service outage. This outage is due to a planned outage by one of our providers, but an unexpected outage for us.

In anticipation of this planned outage, an additional redundant failover system was put in place – however there were problems with the automatic transition when the other uplink went down, causing a failure in our systems.

The planned outage by the one provider may last until 5am. However, automatic failover is still not working properly. If there are more outages, we can manually switch uplinks which takes a little longer but will resolve any long outages.

We will continue to work on the automatic failover system so that in the future, outages due to either provider going down will be a thing of the past.

Thank you!
Andrew Meador

UPDATES:

  • 10:36PM – 12/18/2024
    • Outage began
  • ~11:00PM through ~12:30am – 12/19/2024
    • Outage mostly resolved and down provider up – but could change. Website Systems Status Page updated and modified to provide more information and to also monitor alternate pathways to core systems.

RESOLVED – System Wide Outage

*** Update – 4:15PM 1/11/2024 ***
The upstream provider has resolved their outage! UltraWISP services are back up and running!

There is a known service outage. This outage is not due to a failure in our systems, but upstream in the fiber network between us and the internet at large. There is a reporting error on this page saying it is a Partially Degraded Service – it is not – it is a Full Outage. The St. George – P4 – Gateway is falsely reporting that it is up and running – this is inaccurate.

The upstream carrier that supplies the fiber uplink connection to our network has had had to move their fiber lines due the Corridor H construction in Parsons that has caused this outage. Their planned ETA to finish the scheduled maintenance is between 3:30PM and 4:30PM. This is approximate.

I understand that this is causing everyone difficulties and appreciate your patience while we wait for these repairs to be made.

Thank you! Andrew Meador

RESOLVED – System Wide Outage

*** UPDATE – 5:02AM 5/25/2023 ***
The upstream provider has resolved their outage! UltraWISP services are back up and running!

There is a known service outage. This outage is not due to a failure in our systems, but upstream in the fiber network between us and the internet at large.

The upstream carrier that supplies the fiber uplink connection to our network has had some kind of issue that has caused a wider area outage. They were performing scheduled system maintenance that should NOT have impacted our area, yet did. Their planned ETA to finish the scheduled maintenance is 6am – however – as this related outage was not planned – we’ll see. I will post updates in the morning when I know more.

I understand that this is causing everyone difficulties and appreciate your patience while we wait for these repairs to be made.

Thank you!
Andrew Meador

UPDATES:

  • 12.14AM – 5/25/2023
    • Outage began – ETA for resolution is 6AM
  • 5:02AM – 5/25/2023
    • Outage resolved

RESOLVED – St. George – DMP Outage

*** UPDATE – 11:06PM 5/11/2023 ***

The router has been powered up and UltraWISP services for St. George DMP are back up and running!

There is a known service outage. This is an unplanned outage for the southern area of St. George (DMP Point of Presence) related to an accidental shutdown of the Point of Presence router during system maintenance.

Apologies for any inconvenience this has caused.

Thank you!
Andrew Meador

RESOLVED – System Wide Outage

*** UPDATE – 12:35PM 5/11/2023 ***

The upstream fiber line has been repaired! UltraWISP services are back up and running!

There is a known service outage. This outage is not due to a failure in our systems, but upstream in the fiber network between us and the internet at large.

The upstream carrier that supplies the fiber uplink connection to our network has had some kind of issue that has caused a wider area outage.

I understand that this is causing everyone difficulties and appreciate your patience while we wait for these repairs to be made.

Thank you!
Andrew Meador

UPDATES:

  • 11:04AM – 5/11/2023
    • Outage began
  • 11:44AM – 5/11/2023
    • Upstream fiber provider has wide area outage. Their ETA on repairs is noon.

RESOLVED – System Wide Outage

*** UPDATE – 10:15PM 2/8/2023 ***
The upstream fiber line has been repaired! UltraWISP services are back up and running!

There is a known service outage. This outage is not due to a failure in our systems, but upstream in the fiber network between us and the internet at large.

The upstream carrier that supplies the fiber uplink connection to our network has had some kind of incident that has damaged their fiber lines. They already have crews on site addressing this issue. This is their current statement:

  • “We are aware of a service interruption affecting customers in Davis, WV and surrounding areas [Davis, Hambleton, Hendricks, Parsons, Montrose, and Romney] due to damage to our fiber facilities.

    While our crews are working as quickly as possible to complete the necessary repairs to restore service, the repair of fiber optics is a complex and time-consuming process.

    We know that you depend on [our] services, so we regret the inconvenience this has caused, and we are working to restore services as soon as possible.”

I also understand that this is causing everyone difficulties and appreciate your patience while we wait for these fiber lines to be repaired.

Thank you!
Andrew Meador

UPDATES:

  • 3:45PM – 2/8/2023
    • ETA on fiber repairs is 10PM tonight
  • 10:05PM – 2/8/2023
    • ETA on fiber repairs is 2AM 2/9/2023