Problem

Increased number of errors in production. Some clients were affected for 1 hour.

Action

During investigation, the team noticed a recent deployment which consisted of several database migrations. Breaking and non-breaking migrations executed in wrong order caused columns in Account and User tables to become out of sync. Team manually copied correct values from obsolete columns into the new columns.

Causes

When data migration consists of column removal it would be split into several data migrations. Non-breaking for altering triggers which would be executed immediately and breaking for drop column operations which would be scheduled. That time gap leads to out-of-sync data and errors.

Solutions

We will adjust migration logic to have the trigger modifications together with the column removal in the same migration.