MEWS is slow
Incident Report for Mews
Postmortem

Problem

On 2022-09-26 between 12:38 and 13:35 UTC users experienced degraded response times of the Mews application, approximately 4 times slower than usual.

Action

  • We identified a regressed query indicating selection of a suboptimal execution plan, which was causing high load on the database.
  • We performed mitigating steps to use a better execution plan.

Causes

A query regression occurred when the SQL query optimizer selected a suboptimal execution plan. This caused high resource utilization of our database.

Solutions

As this process is internal to the database engine it does not necessarily relate to any application change.
We are going to implement additional safeguards and keep monitoring on the database level to make sure that a database query regression does not impact the whole application.

Posted Oct 19, 2022 - 12:24 CEST

Resolved
The issue is resolved. We apologize for any inconvenience it may have caused.
Posted Sep 26, 2022 - 16:48 CEST
Monitoring
Mitigation steps performed on the database were successful and we are observing performance improvements.
Posted Sep 26, 2022 - 15:39 CEST
Investigating
We are currently experiencing a higher than normal load on database, and may be causing pages in Mews application to be slow or unresponsive.

We are investigating the cause and will provide an update as soon as possible.
Posted Sep 26, 2022 - 15:31 CEST
This incident affected: Operations, Guest Journey, Business Intelligence, Payments, Open API, and Marketplace.