Reservations coming from channel managers are missing.

Incident Report for Mews

Postmortem

Problem

On 17th May 2025, we experienced an issue in our production system that caused the unintentional removal of key services used for the asynchronous processing of reservations and payments. As a result, around 12,000 reservation messages—particularly from channel partners—were lost or delayed. Some terminal payment events were also affected during this time.

Action

Once the issue was identified, we:

  • Reversed the change that caused the problem.
  • Rebuilt the services responsible for handling reservations and payments.
  • Switched to a backup method for processing reservations to minimize further disruption.
  • Notified affected customers and partners, advising them to reprocess any missing data from the affected period.

Causes

This issue was caused by a configuration error in a system update, which mistakenly removed important data services. The problem was not caught during the review, partly because of how the system presented the changes. It also deployed automatically to all environments, including production, making the impact immediate.

Solutions

To prevent this from happening again, we are:

  • Adding stronger checks and alerts in the asynchronous processing system before any large-scale system changes are made.
  • Improving how system changes in asynchronous processing are previewed to ensure potential issues are clearly flagged during our review process.
  • Adjusting how we deploy updates across environments to enable safer testing of changes.
  • Strengthening our system’s ability to fall back to other processing methods if something goes wrong.
  • Enhancing monitoring and alerting to catch similar problems faster.
Posted May 27, 2025 - 15:20 CEST

Resolved

Channel manager reservation processing is fully operational.
Investigation of missing reservations will follow.
Posted May 18, 2025 - 20:00 CEST

Identified

New reservations are now being created in MEWS.
We are investigating if there any lost reservations and if so how many.
Posted May 18, 2025 - 18:50 CEST

Investigating

Reservations created in OTA and channel managers are not created in Mews.
We are investigating the root cause.
Posted May 18, 2025 - 18:19 CEST
This incident affected: Mews Payments and Mews Marketplace.