Login issue on Mews POS app

Incident Report for Mews

Postmortem

Problem

On 17 February 2025, increased levels of TLS handshake errors were detected during invoice submissions to the Croatian tax agency. A hotfix was released to address the TLS issue, but it inadvertently changed a secret key during a configuration clean-up. This change rendered all logged-in user sessions invalid, causing users to receive "Token invalid" errors and be logged out of the application. The incident affected all currently logged-in users on both Android and web admin platforms, necessitating re-authentication.

Action

As it was a configuration change with an immediate domino effect on all active sessions, all actions were retroactive. A hotfix was deployed that added fallback logic to attempt access token validation with the old key in case of an error. This is a preparatory step for supporting robust key rotation in the future.

Causes

The incident was influenced by both direct and indirect causes. The failure to properly configure the secret key led to immediate user session invalidation. Detection of the incident was delayed due to suppressed validation errors and the absence of alarms for surges in these errors. The resolution was hindered by the timing of the incident and the decision to avoid further user frustration by not invalidating sessions once again during peak hours.

Solutions​

Dealing with sensitive credentials poses significant challenges. Any operation involving keys now must be confirmed and signed off by at least two individuals, aligning with industry’s best practices. Additional tooling to validate the presence of all secrets will be implemented to further safeguard from this incident re-ocurring in the future.

Posted Apr 11, 2025 - 09:00 CEST

Resolved

Due to an unforeseen issue that has been detected and fixed, all login sessions need to be restarted.
All users who experience the "Invalid token" error message are advised to clear the app data from the Android device in order to log in again.

Help center article:
https://help.mews.com/s/article/Troubleshooting-slow-systems-in-Mews-POS?language=en_US#h6

A hotfix was rolled out, enabling devices with no active sessions between 18:00 and 20:00 to log in safely.
Posted Feb 18, 2025 - 17:30 CET