On 17 February 2025, increased levels of TLS handshake errors were detected during invoice submissions to the Croatian tax agency. A hotfix was released to address the TLS issue, but it inadvertently changed a secret key during a configuration clean-up. This change rendered all logged-in user sessions invalid, causing users to receive "Token invalid" errors and be logged out of the application. The incident affected all currently logged-in users on both Android and web admin platforms, necessitating re-authentication.
As it was a configuration change with an immediate domino effect on all active sessions, all actions were retroactive. A hotfix was deployed that added fallback logic to attempt access token validation with the old key in case of an error. This is a preparatory step for supporting robust key rotation in the future.
The incident was influenced by both direct and indirect causes. The failure to properly configure the secret key led to immediate user session invalidation. Detection of the incident was delayed due to suppressed validation errors and the absence of alarms for surges in these errors. The resolution was hindered by the timing of the incident and the decision to avoid further user frustration by not invalidating sessions once again during peak hours.
Dealing with sensitive credentials poses significant challenges. Any operation involving keys now must be confirmed and signed off by at least two individuals, aligning with industry’s best practices. Additional tooling to validate the presence of all secrets will be implemented to further safeguard from this incident re-ocurring in the future.