Unable to reach Mews due to Azure Networking issues

Incident Report for Mews

Postmortem

Problem

Customers were unable to access Mews on 2023-01-25 between 07:08 UTC and 9:30 UTC. During this period connecting to Mews was very slow, and in some cases Mews was unable to load.

Causes

There were global networking issues on the side of our cloud provider Microsoft Azure. Please see the detailed report published by Microsoft: Post Incident Review for Azure Networking – Global WAN issues (Tracking ID VSG1-B90).

Solutions

In addition to the solutions provided by Microsoft, we are planning to enhance our monitoring so we can detect such issues faster and provide a timely report.

Posted Feb 10, 2023 - 16:39 CET

Resolved

Our cloud provider Microsoft Azure confirmed full recovery, with the final networking equipment recovering at 09:35 UTC.
More details on the Azure incident can be found on https://status.azure.com/en-us/status/history/.
Thank you for your patience and we apologize for any inconvenience this may have caused.

Posted Jan 25, 2023 - 12:07 CET

Monitoring

Our cloud provider Microsoft Azure identified a recent networking update and took steps to rollback the recent changes. We are seeing signs of recovery and continue to actively monitor the situation.
You can also check out the Microsoft Azure status page (https://status.azure.com/) for more details and recent updates on this issue.
Thank you for your patience and we apologize for any inconvenience.

Posted Jan 25, 2023 - 11:07 CET

Update

We have identified that the issue is on the side of our cloud provider. Microsoft Azure is currently experiencing a global networking issue which is impacting our availability.
They are working on resolving this issue, and we're monitoring the situation and will provide further updates. We are seeing improvement in client response time and connectivity in the past 30minutes.
You can also check out the Microsoft Azure status page (https://status.azure.com/) for more details and recent updates on this issue.
Thank you for your patience.

Posted Jan 25, 2023 - 10:28 CET

Update

We have identified that the issue is on the side of our cloud provider. Microsoft Azure is currently experiencing a global networking issue which is impacting our availability.
They are working on resolving this issue, and we're monitoring the situation and will provide further updates.
You can also check out the Microsoft Azure status page (https://status.azure.com/) for more details and recent updates on this issue.
Thank you for your patience.

Posted Jan 25, 2023 - 09:41 CET

Identified

We have identified that the issue is on the side of our cloud provider Microsoft Azure. We are working with them to solve this and will provide an update as soon as possible. Thank you for your patience.

Posted Jan 25, 2023 - 09:24 CET

Investigating

We are experiencing a degraded performance of the system. Some pages may be slow or unresponsive.
We're investigating the cause and will provide an update as soon as possible.
Thank you for your patience.

Posted Jan 25, 2023 - 09:00 CET

This incident affected: Operations, Guest Journey, Business Intelligence, Payments, Open API, and Marketplace.