Problem

On 2022-09-19 between 12:15 and 13:47 UTC, all reservations coming into Mews through Channel managers stopped being processed, due to invalid data in our database.

Action

After the issue was detected, we reverted Mews application to the previous version and restore database data to a valid state.

Causes

There were several contributing factors, but the main issue was that we weren't prepared to add reservation sources from multiple, parallel transactions occurring in the system and thus we added the same source multiple times. This had the knock-on effect of causing failure in the processing of all subsequent reservations.

Solutions

1. We have corrected the handling of reservation sources from parallel transactions.
2. We are creating a series of tests to simulate this production environment scenario.
3. We are adding safety measures to ensure that only data in a valid state will be saved to the database.