Articles in this section

See more

Rackspace issues affecting publications [Nov 28th, 2023] (RESOLVED)

Avatar
Joe Turone
  • Updated
Follow

Summary:

We believe we have now fully mitigated the recent incident affecting new content uploads to Signagelive from 17th November 2023 onwards. This was due to an issue with our cloud provider. Should issues persist, please get in touch with our Support Team as any outstanding issues will be handled on a case-by-case basis. The progression of this matter can be tracked below and is considered a Priority 1 issue.

Additional comments:

Whilst the repercussions of this recent incident have been fully mitigated as of December 1st 2023, some content uploaded between 17th November 2023 and when our new resources were put in place may still be impacted. In such cases, please get in touch with our Support Team, additionally - we have provided some additional troubleshooting steps which may assist (depending on the issue faced):

  • If content is not completing download:
    • Layouts: re-save the Layout, this should regenerate any files required for the Layout
    • Videos, Images, Player-Specific Widgets: re-upload the original content to your Signagelive Network once again and re-add the newly updated version to your Playlists
    • Message Manager: Re-upload any images that are not downloading and/or re-save the template and it will regenerate new files

Failing any of the above, please get in touch with our Support Team and we can assist you further with this matter.

[Dec 1st 2023] 16:30 PM (UTC)

As of 1st December 2023 16:30 UTC - we believe we have fully mitigated the issue for new assets and other secondary processes, however, testing is ongoing. Where customers are using assets uploaded since 17th November 2023 15:45 UTC and before our mitigations had been implemented (due to data being in an unexpected state due to API inconsistencies), customer-specific cases continue to be fixed on a case-by-case basis. We are running scans to proactively determine which customers are likely to have issues, however, due to ongoing inconsistencies from Cloud Files, this is a challenging task. Please raise these with Signagelive Support who are resolving issues on a case-by-case basis.

We have a full incident report for this matter available upon request.

----------------------------------------------------------------------------

[Nov 30th 2023] 18:10 PM (GMT)

We are actively addressing challenges arising from an issue with an external service provider. While their service has not yet been fully restored, we've taken steps to minimise the impact on our services.

As of now we have deployed solutions for the following functions -

  • Uploads via the UI
  • Uploads via the API
  • Uploads via the Remote Asset Sync (some API users)
  • Adding from the Marketplace
  • Layouts - Background Colours, Static Text
  • Non Player specific Widgets - see definition below
  • Player-Specific Widgets (offline only) - see definitions below
  • Message Manager - adding new images and editing text - see note below
  • Legacy PC Playlist Writes
  • Publishing of media assets to media players
  • B-Deploy
  • Firmware updates
  • Message Manage asset previews and thumbnails
  • Widget Previews in the UI
  • Exported Reports (Reports, Playlist Export, Player Export)

Given the speed at which these changes have been implemented, it is possible you may see some inconsistencies, however, we are monitoring these and mitigating them as we move through this issue.

We are still working on solutions for the following, and you may see some inconsistencies until we have the redundancy as a result of the dual storage model -

  • Online Widgets - see definition below
  • Layouts -  Background Images (moved back from resolved)
  • File serving from UI and M2M end points

----------------------------------------------------------------------------

[Nov 30th 2023] 12:00 PM (GMT)

Following our previous update in which we adapted our key process to make use of a dual storage solution - as of right now our Development Team have deployed fixes for the following functions:

  • Uploads via the UI
  • Uploads via the API
  • Uploads via the Remote Asset Sync (some API users)
  • Adding from the Marketplace
  • Layouts - Background Colours, Background Images, Static Text
  • Non Player specific Widgets - see definition below
  • Player-Specific Widgets (offline only) - see definitions below
  • Message Manager - adding new images and editing text - see note below
  • Legacy PC Playlist Writes
  • Publishing of media assets to media players
  • B-Deploy
  • Firmware updates
  • Message Manage asset previews and thumbnails

Given the speed at which these changes have been implemented, it is possible you may see some inconsistencies, however, we are monitoring these and mitigating them as we move through this issue.

We are still working on solutions for the following, and you may see some inconsistencies until we have the redundancy as a result of the dual storage model -

  • Online Widgets - see definition below
  • Widget Previews in the UI
  • Exported Reports (Reports, Playlist Export, Player Export)

Thumbnails

Please note that if a thumbnail is not generated for a media asset or does not load, that does not mean it cannot be used in a playlist. If there is metadata (file type, resolution etc) available, then the asset can be added to a playlist and published.

Online / Offline Widget

An offline widget is one that is downloaded to a media player and played from the player's internal storage i.e. most widgets with notable exceptions such as YouTube or widgets published to the Browser Player.

An online widget is one that cannot be downloaded to a player and transparently runs as a static website, notably YouTube or those published to the Browser Player.

Player Specific / Non Player specific widgets

A player-specific widget is one which uses properties from the player, these are unpacked as they are processed by Signagelive and player-specific versions with the player properties embedded. No other changes are made by Signagelive.

A non-player specific widget is one that is published as it is uploaded and does not use properties from the player.

 

Message Manager

When uploading new assets currently, the preview does not load, however, if you click either Add to add the image ‘as-is’ or ‘Edit’ to edit the text layers, it will load and successfully deploy to your player.

----------------------------------------------------------------------------

[Nov 29th 2023] 18:21 PM (GMT)

We have made some significant changes to how media files are stored and processed to mitigate the issues caused by the Rackspace Cloud Files issue whilst Rackspace is working to resolve the issue with their systems.

At a high level, we are storing files and reading in both Rackspace Cloud Files and (now) Amazon S3.

We believe that uploads to Rackspace Cloud Files are actually working they are just not available, and this is inconsistent, varying over time.

We have adapted most of our key processes to both dual store and check both storage locations when loading files, therefore, should either have any read issues, one will return the file; this is a transparent process and fully handled by the Signagelive system.

We have deployed solutions for the following functions:

  • Uploads via the UI
  • Uploads via the API
  • Uploads via the Remote Asset Sync (some API users)
  • Adding from the Marketplace
  • Layouts - Background Colours, Static Text
  • Non Player specific Widgets - see definition below
  • Player-Specific Widgets (offline only) - see definitions below
  • Message Manager - adding new images and editing text - see note below
  • Legacy PC Playlist Writes
  • Publishing of media assets to media players

Given the speed at which these changes have been implemented, it is possible you may see some inconsistencies, however, we are monitoring these and mitigating them as we move through this issue.

We are still working on solutions for the following, and you may see some inconsistencies until we have the redundancy as a result of the dual storage model -

  • Thumbnail generation
  • Layouts - Background images
  • Online Widgets - see definition below
  • Message Manage asset previews
  • Widget Previews in the UI
  • B-Deploy
  • Firmware updates

Thumbnails

Please note that if a thumbnail is not generated for a media asset or does not load, that does not mean it cannot be used in a playlist. If there is metadata (file type, resolution etc) available, then the asset can be added to a playlist and published.

 

Online / Offline Widget

An offline widget is one that is downloaded to a media player and played from the player's internal storage i.e. most widgets with notable exceptions such as YouTube or widgets published to the Browser Player.

An online widget is one that cannot be downloaded to a player and transparently runs as a static website, notably YouTube or those published to the Browser Player.

 

Player Specific / Non Player specific widgets

A player-specific widget is one which uses properties from the player, these are unpacked as they are processed by Signagelive and player-specific versions with the player properties embedded. No other changes are made by Signagelive.

 

A non-player specific widget is one that is published as it is uploaded and does not use properties from the player.

 

Message Manager

When uploading new assets currently, the preview does not load, however, if you click either Add to add the image ‘as-is’ or ‘Edit’ to edit the text layers, it will load and successfully deploy to your player.

----------------------------------------------------------------------------

[Nov 29th 2023] 16:34 PM (GMT)

We would like to thank everyone who is impacted by this issue for their patience and understanding. Rest assured that our Development Team is doing everything possible to complete this mitigation of the cloud provider issue, but this is not a small undertaking. Our Support Team is working to ensure that all customers are kept informed of the progress, and provide assistance on their tickets.

We would like to also make the following clarification about this issue:

Thumbnails:

We are aware that a symptom of this issue is that Thumbnails within your Signagelive asset library may not be generating and therefore visually this will represent to you there is an underlying issue. Whilst that is true in terms of the thumbnail generation, we wish to make explicitly clear that these files are still processing and deployable to your Signagelive Players.

A lack of thumbnail does not mean you cannot schedule and/or deploy your content.

----------------------------------------------------------------------------

[Nov 29th 2023] 14:07 PM (GMT)

Our Development Team have begun deploying updates to begin mitigating the Rackspace issues. This is a large multi-step process, so you may notice slower speeds when navigating the user interface while these updates are taking place. Please notify our Support Team of any issues you experience, our Support Team will also proactively reach out if we notice any further issues on your behalf.

As and when we make progress with this one, we'll update this article accordingly.

----------------------------------------------------------------------------

[Nov 29th 2023] - 10:57 AM (GMT)

The issue is still ongoing and is a priority 1 item. Our development team are working on putting additional resources in place to mitigate the issue whilst we work with our cloud provider to resolve this. Following some additional reports, this issue may materialize in the following manners:

  • Asset Uploads: Newly added assets may not see a thumbnail be generated.
  • Publishing: Players will be connecting but content schedules may appear to not be downloading. This will be represented by an Orange Progress Bar within your Signagelive User interface. Other symptoms related may be old content continuing to be shown, or seeing the default ‘Green Tick’ asset.
  • Widget Error: A “Resource not Found” error message being displayed on the screen or within the User Interface.
  • Widgets: Widgets that have had their Preferences updated after the 17th November may be encountering issues.
  • MRSS Feeds: Feeds updated after the 17th November may be encountering issues.

----------------------------------------------------------------------------

[Nov 28th 2023]

We are aware of an ongoing issue in which media assets are not able to be downloaded to Players - this has resulted in black screens in some instances and old schedules being shown in others. This may be affecting all customers, although we have only had a small number of reports at present.

This is due to an issue with Rackspace, our cloud provider to which we upload and store uploaded media assets. The results are that uploaded assets that are affected are not actually being uploaded to our cloud storage, resulting in Signagelive Players being unable to download these assets if published to a Player.

This issue has been known by Rackspace since November 17th, and you can view the status of their issue here.

  • We are currently performing a review of all Media Assets uploaded to Signagelive since the 17th, to see which are not available.
  • If any assets on your network are impacted, you will be notified by the support team directly.
  • If you have not published any new assets since Nov 17th, then you may not be affected.
  • As this is an ongoing issue with our provider, we will continue to review newly uploaded assets until we are confident the issue is resolved.

 

 

Related articles

Comments

0 comments

Article is closed for comments.