MEWS is slow

Incident Report for Mews

Postmortem

Problem

On 2022-09-26 between 12:38 and 13:35 UTC users experienced degraded response times of the Mews application, approximately 4 times slower than usual.

Action

We identified a regressed query indicating selection of a suboptimal execution plan, which was causing high load on the database.
We performed mitigating steps to use a better execution plan.

Causes

A query regression occurred when the SQL query optimizer selected a suboptimal execution plan. This caused high resource utilization of our database.

Solutions

As this process is internal to the database engine it does not necessarily relate to any application change.
We are going to implement additional safeguards and keep monitoring on the database level to make sure that a database query regression does not impact the whole application.

Posted Oct 19, 2022 - 12:24 CEST

Resolved

The issue is resolved. We apologize for any inconvenience it may have caused.

Posted Sep 26, 2022 - 16:48 CEST

Monitoring

Mitigation steps performed on the database were successful and we are observing performance improvements.

Posted Sep 26, 2022 - 15:39 CEST

Investigating

We are currently experiencing a higher than normal load on database, and may be causing pages in Mews application to be slow or unresponsive.

We are investigating the cause and will provide an update as soon as possible.

Posted Sep 26, 2022 - 15:31 CEST

This incident affected: Mews Operations, Mews Guest Experience, Mews Business Intelligence, Mews Payments, Mews Open API, and Mews Marketplace.