From 14:33 UTC on 27 October until 10:26 UTC on 28 October, guest messaging was unavailable for some properties. Guest Messenger showed "Guest messaging is being set up" instead of the inbox, and messages from Expedia guests were not delivered. Every Expedia message from that period was delivered afterwards, delayed by up to 19 hours. No messages were lost.
We identified the cause on the morning of 28 October. We updated the two affected services with a valid credential and restarted them, which restored guest messaging by 10:26 UTC. We then updated every other service that uses the same credential, and confirmed all delayed Expedia messages were delivered.
We were rotating credentials for our feature flag service, as remediation for an internal security incident contained to those credentials. Many of our services use them to load their feature configuration. The old credentials stayed valid for six days so every service had time to switch. Two services that guest messaging depends on had not switched when the old credentials expired on 27 October. When they next restarted, they could not load their configuration and fell back to default settings, which turn guest messaging off. Our monitoring raised an alert soon after the problem started, but it did not reach an on-call engineer, so we only learned of the problem from customer reports the next morning.