A number of clients were experiencing an outage in the processing of availability, price, and restrictions updates through the channel managers. Later many other clients started experiencing an outage in processing incoming reservations as well.
We increased the processing capacity during the investigation. We ran a job that did fix the processing of availability, price, and restrictions updates, but broke the reservation processing. We found a pattern in the unprocessed incoming reservations and manually fixed it in production.
The change in our processing jobs was not automatically applied and the reset to apply the change broke the setting of our processing jobs.
We will fix our setting for the processing job and create an automatic test for them. We will make queue monitoring more accurate and easier to identify errors. This will also include automatic early notification for our team