Problem

On 2022-09-26 between 12:38 and 13:35 UTC users experienced degraded response times of the Mews application, approximately 4 times slower than usual.

Action

• We identified a regressed query indicating selection of a suboptimal execution plan, which was causing high load on the database.
• We performed mitigating steps to use a better execution plan.

Causes

A query regression occurred when the SQL query optimizer selected a suboptimal execution plan. This caused high resource utilization of our database.

Solutions

As this process is internal to the database engine it does not necessarily relate to any application change.
We are going to implement additional safeguards and keep monitoring on the database level to make sure that a database query regression does not impact the whole application.