Issues with rooms

Incident Report for Mews

Postmortem

Problem

A deployment to switch usage of database tables storing room data was negatively impacted by a previous bug which resulted in the data not being mirrored correctly to the new table. This in turn impacted information stored about room state as well as parent data, creating a scenario wherein spaces could be set to the wrong state and a wrong parent.

In addition, there was another bug introduced in this deploy due to which any change of room through the application UI deleted the room parent. It was therefore not possible to manually reassign the parent.

This incorrect parent/ child setup produced colliding reservations and potential overbookings for properties using nested spaces.

Actions

A team was established to work on the issues. While one part of the team restored the room status data from backup, the other part fixed problems causing the corrupted parent data.
A hotfix was deployed, to fix the root cause of the data loss and restore parent data.
The Mews support team contacted properties affected by the overbooking.

Causes

Wrong configuration of the database table migration.
Long data recovery time led to delayed resolution.

Solutions

We are evaluating reimplementation of some of the backend sections, to make it easier to catch such bugs earlier in development.
We will implement a safer way of handling migration of database structures.
We will update our processes with a guide on how to handle data corruption, to reduce future reaction time in such scenarios.

Posted Jul 12, 2022 - 10:17 CEST

Resolved

We restored all corrupted data of nested spaces based on the morning backup. We would like to ask you to check your timeline and space configuration. Please be aware the bug led to some colliding reservations created for your nested spaces structure and our support team will get in touch with the affected clients and help to identify them.
We sincerely apologize for the inconvenience the bug caused. We are going to evaluate all the steps that led to this situation as part of our postmortem process to avoid such incidents happening in the future.

Posted Jun 14, 2022 - 23:54 CEST

Update

The hotfix of assigning parent rooms to the spaces has been deployed. We are recovering the corrupted data prior to the deployment containing the bug which will take a few hours. If your operation is blocked by the wrong nested spaces configuration, please update the space structure manually by navigating to Menu-Settings-Property-Space configuration-Spaces - click on the space and select the parent room.
See the help page describing the parent space assignment on https://help.mews.com/s/article/how-to-set-up-a-parent-room?language=en_US
Note: Only users with the ''Create and manage space categories'' privilege are able to proceed with manual updates.

Posted Jun 14, 2022 - 21:40 CEST

Update

We found a bug with nested spaces and we are deploying the hotfix. In the meantime to mitigate the possible overbooking of those spaces, please set up the restrictions for them. We will update the status page once the hotfix is deployed so you can start setting back the nested spaces to the correct state. In the meantime, we are also working on restoring the corrupted data and fix nested spaces in batch.

Posted Jun 14, 2022 - 19:30 CEST

Identified

We have found a number of inconsistent rooms that were updated before we managed to fix the data. We are in the process of restoring the data from a backup, please do not update rooms in the meanwhile.

Posted Jun 14, 2022 - 19:03 CEST

Monitoring

We have fixed the underlying issues and we're monitoring the results.

Posted Jun 14, 2022 - 18:33 CEST

Identified

We have identified the last issue that was concerning parent/child space configuration and we're working on the fix. Please avoid manually fixing the data to avoid potential interference.

Posted Jun 14, 2022 - 18:18 CEST

Update

We have found and fixed the root cause for inconsistent data in room states and we're continuing our investigation.

Posted Jun 14, 2022 - 18:06 CEST

Investigating

We have identified a number of issues with rooms, we are investigating the cause. This includes unexpected behavior with room allocation, sudden change of room states, parent/child space configuration and the timeline in general.

Posted Jun 14, 2022 - 17:28 CEST

This incident affected: Operations.