UI Service and Outage Issues - 08/02/2022 - [Completed]

SUMMARY

Rackspace Update [13:48PM]
The associated Rackspace DWDM network provider is actively investigating as a priority and as of 8 March 2022 13:29 UTC are starting to see an improvement in available capacity with environment checkouts currently in progress.

PAST UPDATES

8TH MARCH 13:03 PM GMT (21:03 PM SGT):

Our testing and monitoring is showing that services are recovering. At this stage we are contacting Rackspace to get an official update as they can take some time to filter through the support tickets and public status page, before we re-enable services so we don't have a false start for Signagelive users.

As soon as we know more we will post new updates here. Sorry for any inconvenience caused.

8TH MARCH 11:46 AM GMT (19:46 PM SGT):

Unfortunately we do not have any further updates at this stage. We are still waiting on the fibre provider to resolve the issue. We have had it confirmed to us that engineers are on site and working on it as a priority.

We internally discussed if there are any temporary solutions we can put in place, however this is highly complex as we do not know with complete certainty which components are affected, and which can and cannot communicate together. We have no control over this based on the current issues, and therefore at this stage are holding off on any temporary solutions.

8TH MARCH 09:30 AM BST (17:30 PM SGT):

Given the connectivity issues, Signagelive has been taken offline. When logging in, the maintenance page will be updated with the following:

====================
"We have taken Signagelive offline as we have been affected by major connectivity issues at and between our data centres provided by Rackspace which is causing intermittent stability issues.

The main fibre cable has been damaged which is being repaired and two of the three other redundant fibre connections are unavailable.

We are working with Rackspace to bring services online as quickly as possible, however as this is a core network infrastructure issue we have to wait for that underlying infrastructure to be restored.

We will provide status updates here, and Rackspace’s Status page provides further information relating to the core issue.
====================

As soon as we know more we will post new updates here. Sorry for any inconvenience caused.

8TH MARCH 08:00 AM BST (16:00 PM SGT):

Based on recent communication with the data centre, they inform us they are still working on fixing the damaged fibre cable but unfortunately are still unable to provide an ETA for a resolution / workaround at this stage.

8TH MARCH 03:15 AM BST (11:15 AM SGT):

There is a network issue in the data centre that hosts the Signagelive servers. Report below from the data centre:

On 7 March 2022, at 22:17 BST, some customers may be experiencing intermittent issues including potential latency, packet loss or failed connectivity. Engineers determined that an edge device is showing dark fibre faults and three out of four dark fibre circuits are down. This is also impacting connectivity between LON3 and LON5. Rackspace is contacting the vendor to assist in investigating.

A fix for this issue is reliant on the data centre's ability to either put in place a workaround or permanent fix. In the mean time we have rebooted the Signagelive UI servers and have seen an improvement in performance.

ISSUE SUMMARY

We received reports of access issues to the Signagelive user-interface (UI), with an error message appearing on-screen after attempting to log-in. General UI slowness is also being experienced.

FOR MORE INFORMATION

For current system status information about Signagelive, please check out our system status page. During an issue, you can also receive status updates by following Beamer Posts within the user interface. The summary of our investigation will be posted here when the issues has ended. If you have any concerns, questions, wish to report further issues or are looking for updates - please don't hesitate to contact our Support Team.