Degraded system performance after deployment

Incident Report for Mews

Postmortem

Problem

On September 4 at 13:41 UTC, the end-user latency across Mews products increased and stayed elevated until 13:59 UTC.

Causes

We deployed a change that led, by updating a high-traffic table schema, to an excessive database utilization.

Action

We rolled the deployment back, which resolved the issue.

Solutions

We identify opportunities to avoid doing the problematic database changes in some situations and to mitigate their impact in situations where they need to be executed.

Posted Sep 09, 2024 - 16:17 CEST

Resolved

The issue is resolved and we do not expect any further latency issues from this deployment update.
Posted Sep 04, 2024 - 17:16 CEST

Monitoring

We've reverted a deployment and are monitoring the situation.
Posted Sep 04, 2024 - 16:30 CEST

Identified

We've spotted that after recent deployment system performance is degraded. We're currently investigating the issue, and will provide an update soon.
Posted Sep 04, 2024 - 16:04 CEST
This incident affected: Operations, Guest Journey, Business Intelligence, Payments, Open API, Marketplace, and Aircall Inbound calls.