Problem

On 9 September between 21:07 and 21:25 CEST, the Mews platform experienced a performance degradation lasting about 18 minutes. About half of requests to the application and API timed out, including sign-in. Customers saw slow or unresponsive pages and intermittent errors. Some operations kept working, and no data was lost.

Action

Our automated monitoring detected the issue within seven minutes, and our support team was engaged within minutes to help affected customers. We kept customers informed through our status page and monitored until the service fully recovered at 21:25 CEST. We opened a case with our cloud provider the same evening.

Causes

A routine access change to our production database caused the incident. Because of how our tooling re-applied existing permissions, one small change became a large batch of permission updates. That batch caused the database replica that serves read traffic to stall. Our cloud provider has confirmed that the replica should have recovered far sooner than it did, and is working on improvements to how it recovers from this situation. The replica recovered on its own after about 18 minutes, and operations that write data were largely unaffected.

Solutions

• We are changing our database tooling to apply only the specific change being made, rather than re-applying every existing permission.
• We are adding timing, announcement and approval controls for production database changes such as this one.
• We are making infrastructure changes visible to our incident tooling, so similar changes are identified immediately.
• We have added alerting on this type of database change, and are improving alerting on database connection errors.
• We are working with our cloud provider on faster recovery from this replica behaviour, and adding a fallback so read traffic can continue if the replica is unavailable.