Mews is slow

Incident Report for Mews

Postmortem

Problem

On 2022-09-06 between 10:28 and 12:06 UTC we experienced very slow response times of the Mews application, spiking up to 8s.

Action

We identified a group of regressed queries that were causing high load on database and forced better execution plans for them. There was a full database backup running at that time, which made the mitigation slower.

Causes

There was a combination of several factors. One of them being full database backup which runs for a long time due to the database size. It coincided with application deployment, which introduced database changes and contributed to significant regression for a group of SQL queries.

Solutions

We are going to reschedule the regular deployments so that they do not coincide with database backup.

Posted Sep 19, 2022 - 12:28 CEST

Resolved

The issue is resolved. We apologize for any inconvenience it may have caused.

Posted Sep 06, 2022 - 15:52 CEST

Update

We are observing a full system recovery. We continue to monitor the situation and will provide further updates.

Posted Sep 06, 2022 - 14:16 CEST

Monitoring

Mitigation steps performed on the database were successful and we are observing performance improvements. We will provide an update as soon as possible.

Posted Sep 06, 2022 - 14:05 CEST

Identified

We have identified some regressed queries on database and are performing mitigation steps. We will provide an update in 1 hour or as new events occur.

Posted Sep 06, 2022 - 13:48 CEST

Update

We are currently experiencing a higher than normal load on database, which may be causing pages in Mews application to be slow or unresponsive.

We are investigating the cause and will provide an update as soon as possible.

Posted Sep 06, 2022 - 12:49 CEST

Investigating

We are currently experiencing a higher than normal load on database, and may be causing pages in Mews application to be slow or unresponsive.

We are investigating the cause and will provide an update as soon as possible.

Posted Sep 06, 2022 - 12:34 CEST

This incident affected: Operations, Guest Journey, Business Intelligence, Payments, Open API, and Marketplace.