db:migrate:multi-version-upgrade job

• Introduced in GitLab 16.11.

This job runs on the test stage of a merge request pipeline. It validates that migrations pass for multi-version upgrade from the latest required upgrade stop to the author's working branch. It achieves it by running gitlab:db:configure against PostgreSQL dump created from the latest known GitLab version stop with test data.

The database dump is generated and maintained with PostgreSQL Dump Generator. To seed database with data, the tool uses Data Seeder with bulk_data.rb configuration to seed all factories and uses db:seed_fu to seed all db/fixtures. Latest dump is generated automatically in scheduled pipelines for the latest patch release of the required stop.

Troubleshooting

Database reconfigure failures

This failure usually happens due to an actual migration error in your working branch. To reproduce the failure locally follow Migration upgrade testing guidance. It outlines the steps how to import the latest PostgreSQL dump in your local GitLab Development Kit or GitLab Docker instance.

For a real-life example, refer to this failed job.

Broken master

When a new required upgrade stop is added (every three or four milestones), it triggers the build of a new PostgreSQL Dump. In some cases, this might cause the db:migrate:multi-version-upgrade job to fail in master pipeline. For example, if new additional tables are seeded, it helps detect migration errors that might have been missed in older dumps without these seeded tables.

Workflow for the broken master case:

1. Identify the root cause MR by searching for the migration that caused the error. For instance, db/migrate/20240416123401_add_security_policy_management_project_id_to_security_policies.rb in failing job
   • Debug locally if needed as described in the Database reconfigure failures
2. Reach out to the relevant team that introduced the migration to discuss whether the MR should be reverted or if a fix will be worked on
   • If the team isn't available, post about it in the #database Slack channel
3. While a fix or revert is being worked on, master pipeline can be unblocked by disabling the job temporarily by setting DISABLE_DB_MULTI_VERSION_UPGRADE=true in CI/CD Settings page
   • When disabling the job, announce it in the #master-broken Slack channel
4. Add a note to job stability tracking issue#458402
5. Reinstate the job by removing DISABLE_DB_MULTI_VERSION_UPGRADE from CI/CD Settings

Database import failures

If job is failing on setup stage prior to gitlab:db:configure due to external dependencies, the job can be disabled by setting DISABLE_DB_MULTI_VERSION_UPGRADE=true in GitLab project CI variables to unblock the broken master.

Reach out to Self-Managed Platform team to expedite debugging.