Guidance required for migrating Airflow 2.0.0 metadata to Airflow 3.x on AKS

["Guruswamy, Hemanthraj via users" <us...@airflow.apache.org>]

Hi All,

We are currently running a production (live) Apache Airflow instance with the following setup:

Existing Environment:

- Airflow version: 2.0.0
- Platform: Azure Kubernetes Service (AKS 1.23.8)
- Database: Azure Database for PostgreSQL Flexible Server (version 12.22)

We have provisioned a new Airflow environment with the following configuration:

New Environment:

- Airflow version: 3.1.7
- Platform: Azure Kubernetes Service (AKS 1.33.0)
- Database: Azure Database for PostgreSQL Flexible Server (version 17.9)

Requirement:
We need to migrate the metadata (Airflow database) from the existing (Airflow 2.0.0) instance to the new Airflow 3.x instance.

Issue:
When we attempt to export and import (pg_dump /pg_restore) the metadata database from the old PostgreSQL instance to the new one, the Airflow application does not start. It appears that the database schema and metadata objects from the older version are not compatible with Airflow 3.x.

Understanding:
Based on the official documentation, Airflow upgrades require a sequential upgrade path (e.g., 2.0.0 → 2.7 → 3.x) to ensure proper schema migrations.

Clarification Required:
Since we have already provisioned a fresh Airflow 3.x environment, we would like guidance on the following:

1. What is the recommended approach to migrate metadata from Airflow 2.0.0 to Airflow 3.x?
2. Is it mandatory to perform in-place upgrades (2.0.0 → intermediate versions → 3.x) instead of direct database migration?
3. Can we safely migrate metadata without impacting the currently running production Airflow instance?
4. Are there any supported tools, migration utilities, or best practices for this scenario?

Our goal is to set up the new Airflow 3.x instance with the existing metadata while keeping the current production instance unaffected.

We appreciate your guidance on the correct approach.

Thanks in advance,
Hemanth Raj




________________________________________
IMPORTANT - PLEASE READ: This electronic message, including its attachments, is CONFIDENTIAL and may contain PROPRIETARY or LEGALLY PRIVILEGED or PROTECTED information and is intended for the authorized recipient of the sender. If you are not the intended recipient, you are hereby notified that any use, disclosure, copying, or distribution of this message or any of the information included in it is unauthorized and strictly prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and permanently delete this message and its attachments, along with any copies thereof, from all locations received (e.g., computer, mobile device, etc.). To the extent permitted by law, we may monitor electronic communications for the purposes of ensuring compliance with our legal and regulatory obligations and internal policies. We may also collect email traffic headers for analyzing patterns of network traffic and managing client relationships. For further information see our privacy-policy<https://www.iqvia.com/about-us/privacy/privacy-policy>. Thank you.

Re: Guidance required for migrating Airflow 2.0.0 metadata to Airflow 3.x on AKS

[Jens Scheffler <js...@apache.org>]

Hi Hemanth Raj,

this is the users mainling list and not the "free support forum" or 
consulting. Such questions are usually best in the 
Slack/#user-troubleshooting section or if it is an issue then in Github 
discussions as well.

But to help you in short:

Regarding 1) Yes, migrate the Airflow DB to Airflow 2.7.x and then to 
3.x. You probably do not need a full install but can spin-up a Pod with 
the Airflow docker and call "airflow db migrate".

Regarding 2) You can decide. It is open soruce software and the 
operation options and responsibility is fully on your side.

Regarding 3) It is not safe by default. Each migration will make 
structural changes and probably while migrating the old SW will not 
function and the new not as well until completed. But it depends on each 
version bump. 2.x to 3.x you should consider making offline for sure. 
Minor versions might not be critical if only a index is added. But 
please consult the documentaion and details about migration in 
https://airflow.apache.org/docs/apache-airflow/stable/migrations-ref.html

Regarding 4) Yes. Please read 
https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading.html 
and in your case especially 
https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html

General note: I'd propose if you anyway make an upgrade to always pick 
the most recent version. We are at 3.2.1. So 3.1.7 is at the time of 
writing already somewhat outdated. Regular updates are needed to keep 
you stable and secure.

Jens

Note: The mailing list is not free consulting. All statements are given 
without warranty

On 15.05.26 07:08, Guruswamy, Hemanthraj via users wrote:
>
> Hi All,
>
> We are currently running a production (live) Apache Airflow instance 
> with the following setup:
>
> *_Existing Environment:_*
>
> - Airflow version: 2.0.0
>
> - Platform: Azure Kubernetes Service (AKS 1.23.8)
>
> - Database: Azure Database for PostgreSQL Flexible Server (version 12.22)
>
> We have provisioned a new Airflow environment with the following 
> configuration:
>
> *_New Environment:_*
>
> - Airflow version: 3.1.7
>
> - Platform: Azure Kubernetes Service (AKS 1.33.0)
>
> - Database: Azure Database for PostgreSQL Flexible Server (version 17.9)
>
> *Requirement:*
>
> We need to migrate the metadata (Airflow database) from the existing 
> (Airflow 2.0.0) instance to the new Airflow 3.x instance.
>
> Issue:
>
> When we attempt to export and import (pg_dump /pg_restore) the 
> metadata database from the old PostgreSQL instance to the new one, the 
> Airflow application does not start. It appears that the database 
> schema and metadata objects from the older version are not compatible 
> with Airflow 3.x.
>
> *Understanding:*
>
> Based on the official documentation, Airflow upgrades require a 
> sequential upgrade path (e.g., 2.0.0 → 2.7 → 3.x) to ensure proper 
> schema migrations.
>
> *Clarification Required:*
>
> Since we have already provisioned a fresh Airflow 3.x environment, we 
> would like guidance on the following:
>
> 1. What is the recommended approach to migrate metadata from Airflow 
> 2.0.0 to Airflow 3.x?
>
> 2. Is it mandatory to perform in-place upgrades (2.0.0 → intermediate 
> versions → 3.x) instead of direct database migration?
>
> 3. Can we safely migrate metadata without impacting the currently 
> running production Airflow instance?
>
> 4. Are there any supported tools, migration utilities, or best 
> practices for this scenario?
>
> Our goal is to set up the new Airflow 3.x instance with the existing 
> metadata while keeping the current production instance unaffected.
>
> We appreciate your guidance on the correct approach.
>
> Thanks in advance,
>
> Hemanth Raj
>
>
>
> ________________________________________
> *IMPORTANT* - PLEASE READ: This electronic message, including its 
> attachments, is CONFIDENTIAL and may contain PROPRIETARY or LEGALLY 
> PRIVILEGED or PROTECTED information and is intended for the authorized 
> recipient of the sender. If you are not the intended recipient, you 
> are hereby notified that any use, disclosure, copying, or distribution 
> of this message or any of the information included in it is 
> unauthorized and strictly prohibited. If you have received this 
> message in error, please immediately notify the sender by reply e-mail 
> and permanently delete this message and its attachments, along with 
> any copies thereof, from all locations received (e.g., computer, 
> mobile device, etc.). To the extent permitted by law, we may monitor 
> electronic communications for the purposes of ensuring compliance with 
> our legal and regulatory obligations and internal policies. We may 
> also collect email traffic headers for analyzing patterns of network 
> traffic and managing client relationships. For further information see 
> our privacy-policy 
> <https://www.iqvia.com/about-us/privacy/privacy-policy>. Thank you.