On 2020-08-30 15:35 an issue was reported by our Customer Care team notifying us that customers were not able to enter card details. Our developers were notified at 2020-08-30 15:37.
At 2020-08-30 15:49 we checked the dashboard of our external card storage provider. We saw a lower volume of cards being tokenized and at the same time we saw errors being report in our own logs. Shortly after this, the external provider notified our support that they were dealing with an internet outage.
We started monitoring the errors on the dashboard of our external provider and in our own logs, around 2020-08-30 16:52 the rate of errors started slowing down and by 17:03. The external provider notified us that the issue was resolved. By 2020-08-30 17:12 we updated the statuspage and marked the issue as resolved.
As the outage was caused by an external company, the only action we would take was monitor the situation for any improvements. We did this checking our partners external dashboard and our own error logs.
1. Widespread internet outage for CenturyLink in the US and Europe. (https://www.digitaltrends.com/web/centurylink-internet-service-outage/)
1. Improve logging on frontend, to see if required files from the external provider are loaded correctly.
2. Improve logging on backend, to check for anomalies in the rate of tokenized cards.
3. Display a more userfriendly message in case of an outage at the external provider.