Problem

Degradation of database response time - overall application response time was elevated during 10 min.

Action

Recent changes on the index were manually rolled back.

Causes

To improve performance one existing index was updated. Even though the change seemed to be reasonable it affected the query plan selection in some corner cases. Thus, application response time was affected only 10 min in total during the whole 45 min incident.

Solutions

We will consume SQL diagnostic data and build an alert on top of it to catch query response degradation earlier.