Messaging degradation

Incident Report for Mews

Postmortem

Problem

Messaging clients disconnected between 9:17 and 9:30 UTC.

Action

At 9:17 UTC, messaging clients disconnected, affecting user access to messaging services. The team quickly rolled back to a previous version, restoring service by 9:30 UTC.

Causes

A bug in the deployment process caused an update to be applied incorrectly. This led to messaging client disconnections.

Solutions

The fix was implemented, and the system was reviewed. Messaging services are fully operational, and no further issues were found.

Posted Oct 24, 2024 - 18:05 CEST

Resolved

The problem causing messaging issues has been fully resolved as of 9:30 UTC.

Posted Oct 03, 2024 - 15:26 CEST

Monitoring

The Mews Platform team has identified an issue in the latest release and switched back to previous version at 9:30UTC. We are observing that messaging clients reconnected and continue monitoring the situation.

Posted Oct 03, 2024 - 11:43 CEST

Investigating

We are investigating issues with messaging across the application which started at 9:17 UTC. We will provide an update as soon as possible.

Posted Oct 03, 2024 - 11:36 CEST

This incident affected: Operations.