You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "n-badtke-cg (via GitHub)" <gi...@apache.org> on 2026/08/18 14:07:13 UTC

[I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

n-badtke-cg opened a new issue, #71772:
URL: https://github.com/apache/airflow/issues/71772

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.3.1
   
   ### What happened and how to reproduce it?
   
   We are currently running Airflow 3.3.0 in Kubernetes using the Airflow Helm Chart v1.22 for deployment. All Dags are healthy. The Dags are fetched via Git Dag Bundle.
   
   After deploying, the dag processor messages errors like `Processor for DagFileInfo(rel_path=PosixPath('<filepath_in_git_repo>'), bundle_name='<a_name>', bundle_path=PosixPath('/tmp/airflow/dag_bundles/<a_name>/tracking_repo', bundle_version=None) with PID NNN has been running for 50.01 seconds, exceeding the timeout of 50.00 seconds. Killing it! [airflow.dag_processing.manager.DagFileProcessorManager] loc=manager.py:1621` and the overview table now shows "# Errors" > 0
   
   We have a very basic "airflow_health_check" Dag to verify the Airflow workers Dag runtime:
   
   ```py
   from airflow import XComArg
   from airflow.sdk import dag, task
   from airflow.sdk.exceptions import AirflowFailException
   from datetime import datetime, timedelta
   
   HEALTH_CHECK_MESSAGE: str = "Health check successful"
   HEALTHY: str = "healthy"
   UNHEALTHY: str = "unhealthy"
   
   
   @dag(
       schedule="*/2 * * * *",
       start_date=datetime(2026, 7, 1),
       dagrun_timeout=timedelta(minutes=3),
       is_paused_upon_creation=False,
       catchup=False,
   )
   def airflow_health_check() -> None:
       @task.bash
       def health_check(text: str) -> str:
           return f"echo '{text}'"
   
       @task
       def result_is_healthy(*, result: XComArg, check_against: str) -> bool:
           return str(result) == check_against
   
       @task
       def print_result(is_healthy: bool) -> None:
           print("is_healthy: " + str(type(is_healthy)) + " " + str(is_healthy))
           print(HEALTHY if is_healthy == True else UNHEALTHY)
           if not is_healthy == True:
               raise AirflowFailException()
   
       health_check_result: XComArg = health_check(HEALTH_CHECK_MESSAGE)
       result_is_healthy_bool: bool = result_is_healthy(
           result=health_check_result, check_against=HEALTH_CHECK_MESSAGE
       )
       print_result(result_is_healthy_bool)
   
   
   airflow_health_check()
   ```
   
   With Airflow 3.3.0, the Dag Runs are green.  
   Shortly after the upgrade to 3.3.1, the Dag Runs are stuck in "queued", but the Dag itself is still active.  
   With the Dag processor catching up and now throwing errors, the Dag gets "Deactivated".
   
   ### What you think should happen instead?
   
   It should just work? Idk, do I overlook something?
   
   ### Operating System
   
   Container image apache/airflow:3.3.1-python3.13
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Apache Airflow Provider(s)
   
   amazon, celery, cncf-kubernetes, common-compat, common-io, common-messaging, common-sql, databricks, docker, elasticsearch, fab, ftp, git, google, grpc, hashicorp, http, keycloak, microsoft-azure, mysql, odbc, openlineage, opensearch, postgres, redis, sendgrid, sftp, slack, smtp, snowflake, ssh, standard
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon==9.34.0
   apache-airflow-providers-celery==3.23.1
   apache-airflow-providers-cncf-kubernetes==10.21.0
   apache-airflow-providers-common-compat==1.18.0
   apache-airflow-providers-common-io==1.8.0
   apache-airflow-providers-common-messaging==2.0.4
   apache-airflow-providers-common-sql==2.1.0
   apache-airflow-providers-databricks==7.18.1
   apache-airflow-providers-docker==4.5.9
   apache-airflow-providers-elasticsearch==6.9.0
   apache-airflow-providers-fab==3.8.0
   apache-airflow-providers-ftp==3.15.2
   apache-airflow-providers-git==0.4.2
   apache-airflow-providers-google==22.3.0
   apache-airflow-providers-grpc==3.9.5
   apache-airflow-providers-hashicorp==4.8.0
   apache-airflow-providers-http==6.0.5
   apache-airflow-providers-keycloak==0.8.2
   apache-airflow-providers-microsoft-azure==14.1.0
   apache-airflow-providers-mysql==6.6.1
   apache-airflow-providers-odbc==4.12.3
   apache-airflow-providers-openlineage==2.20.0
   apache-airflow-providers-opensearch==1.12.0
   apache-airflow-providers-postgres==7.0.1
   apache-airflow-providers-redis==4.5.0
   apache-airflow-providers-sendgrid==4.2.4
   apache-airflow-providers-sftp==6.0.1
   apache-airflow-providers-slack==9.10.2
   apache-airflow-providers-smtp==3.0.3
   apache-airflow-providers-snowflake==6.16.0
   apache-airflow-providers-ssh==6.0.1
   apache-airflow-providers-standard==1.17.0
   
   ### Official Helm Chart version
   
   1.22.0 (latest released)
   
   ### Kubernetes Version
   
   v1.35.1
   
   ### Helm Chart configuration
   
   Incomplete, only an excerpt. Please let me know if there might be something other relevant to share.
   
   ```yaml
   airflowVersion: 3.3.1
   config:
     azure_remote_logging:
       remote_wasb_log_container: airflow-remote-logs
     celery_broker_transport_options:
       visibility_timeout: 14400
     core:
       auth_manager: airflow.providers.keycloak.auth_manager.keycloak_auth_manager.KeycloakAuthManager
       test_connection: Hidden
     keycloak_auth_manager:
       realm: application
       requests_pool_size: 20
       server_url: <a_valid_url>
     logging:
       delete_local_logs: "False"
       remote_base_log_folder: wasb://logs
       remote_log_conn_id: airflow-remote-logging-wasb
       remote_logging: "True"
   
   dagProcessor:
     dagBundleConfigList:
     - classpath: airflow.providers.git.bundles.git.GitDagBundle
       kwargs:
         git_conn_id: <a_name_for_conn>
         refresh_interval: 30
         tracking_ref: main
       name: <a_name>
   ```
   
   
   ### Docker Image customizations
   
   Dockerfile:
   ```dockerfile
   ARG AIRFLOW_CONTAINER_VERSION=3.3.0-python3.13
   
   FROM apache/airflow:${AIRFLOW_CONTAINER_VERSION}
   
   USER root
   
   COPY org_ca.pem /tmp/org_ca.pem
   RUN csplit -z -f /usr/local/share/ca-certificates/org_ca_ \
           --suffix-format='%03d.crt' \
           /tmp/org_ca.pem \
           '/-----BEGIN CERTIFICATE-----/' '{*}' \
       && update-ca-certificates \
       && rm /tmp/org_ca.pem
   
   USER airflow
   
   COPY upstream-requirements.txt upstream-requirements.txt
   RUN --mount=type=secret,id=pip-conf,dst=/etc/pip.conf,mode=0440 \
       pip install --no-cache-dir \
       --constraint /home/airflow/constraints.txt \
       --requirement upstream-requirements.txt
   
   # install custom providers and packages in separate pip install command to avoid dependency version clash with constraints
   COPY custom-requirements.txt custom-requirements.txt
   RUN --mount=type=secret,id=pip-conf,dst=/etc/pip.conf,mode=0440 \
       pip install --no-cache-dir \
       --requirement custom-requirements.txt
   
   ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
   ```
   
   upstream-requirements.txt:
   ```
   # Environmernt variable ${AIRFLOW_VERSION} comes from the upstream Airflow image and is resolved by pip
   apache-airflow[cloudpickle,databricks,keycloak,microsoft.azure,otel]==${AIRFLOW_VERSION}
   
   # Airflow components not covered via extras, versions are covered by constraints file
   virtualenv
   ```
   
   custom-requirements.txt:
   ```
   # Environment variable ${AIRFLOW_VERSION} comes from the upstream Airflow image and is resolved by pip
   # Pin Airflow version to prevent pip from changing it during dependency resolution
   apache-airflow==${AIRFLOW_VERSION}
   
   <some propietary providers>
   ```
   
   ### Anything else?
   
   To be honest, I am a bit lost where to look at else. Please let me know :)
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5343978725

   @tirkarthi  Given your commands, I opened a remote shell to the Kubernetes Pod did what you asked for:
   <details>
     <summary>bash commands dump (Click to expand)</summary>
   
     ```bash
     $ kubectl exec -n airflow airflow-dag-processor-66894c6b74-bkp4d -c dag-processor -it -- bash -il
     airflow@airflow-dag-processor-66894c6b74-bkp4d:/opt/airflow$ cd dags
     airflow@airflow-dag-processor-66894c6b74-bkp4d:/opt/airflow/dags$ cat > gh71772.py<< EOF
     > from __future__ import annotations
     
     from datetime import datetime, timedelta
     
     from airflow import XComArg
     from airflow.sdk import dag, task
     from airflow.sdk.exceptions import AirflowFailException
     
     HEALTH_CHECK_MESSAGE: str = "Health check successful"
     HEALTHY: str = "healthy"
     UNHEALTHY: str = "unhealthy"
     
     
     @dag(
         schedule="*/2 * * * *",
         start_date=datetime(2026, 7, 1),
         dagrun_timeout=timedelta(minutes=3),
         is_paused_upon_creation=False,
         catchup=False,
     )
     def gh71772() -> None:
         @task.bash
         def health_check(text: str) -> str:
             return f"echo '{text}'"
     
         @task
         def result_is_healthy(*, result: XComArg, check_against: str) -> bool:
             return str(result) == check_against
     
         @task
         def print_result(is_healthy: bool) -> None:
             print("is_healthy: " + str(type(is_healthy)) + " " + str(is_healthy))
             print(HEALTHY if is_healthy == True else UNHEALTHY)
             if not is_healthy == True:
                 raise AirflowFailException()
     
         health_check_result: XComArg = health_check(HEALTH_CHECK_MESSAGE)
         result_is_healthy_bool: bool = result_is_healthy(
             result=health_check_result, check_against=HEALTH_CHECK_MESSAGE
         )
         print_result(result_is_healthy_bool)
     
     
     gh71772()
     > EOF
     airflow@airflow-dag-processor-66894c6b74-bkp4d:/opt/airflow/dags$ cat gh71772.py
     from __future__ import annotations
     
     from datetime import datetime, timedelta
     
     from airflow import XComArg
     from airflow.sdk import dag, task
     from airflow.sdk.exceptions import AirflowFailException
     
     HEALTH_CHECK_MESSAGE: str = "Health check successful"
     HEALTHY: str = "healthy"
     UNHEALTHY: str = "unhealthy"
     
     
     @dag(
         schedule="*/2 * * * *",
         start_date=datetime(2026, 7, 1),
         dagrun_timeout=timedelta(minutes=3),
         is_paused_upon_creation=False,
         catchup=False,
     )
     def gh71772() -> None:
         @task.bash
         def health_check(text: str) -> str:
             return f"echo '{text}'"
     
         @task
         def result_is_healthy(*, result: XComArg, check_against: str) -> bool:
             return str(result) == check_against
     
         @task
         def print_result(is_healthy: bool) -> None:
             print("is_healthy: " + str(type(is_healthy)) + " " + str(is_healthy))
             print(HEALTHY if is_healthy == True else UNHEALTHY)
             if not is_healthy == True:
                 raise AirflowFailException()
     
         health_check_result: XComArg = health_check(HEALTH_CHECK_MESSAGE)
         result_is_healthy_bool: bool = result_is_healthy(
             result=health_check_result, check_against=HEALTH_CHECK_MESSAGE
         )
         print_result(result_is_healthy_bool)
     
     
     gh71772()
     airflow@airflow-dag-processor-66894c6b74-bkp4d:/opt/airflow/dags$ python
     Python 3.13.14 (main, Aug 12 2026, 08:57:09) [GCC 12.2.0] on linux
     Type "help", "copyright", "credits" or "license" for more information.
     >>>  from airflow.dag_processing.dagbag import DagBag
     ...
     KeyboardInterrupt
     >>> from airflow.dag_processing.dagbag import DagBag
     2026-08-19T14:50:49.459701Z [warning  ] Both the standard OpenTelemetry environment variables and the Airflow OpenTelemetry configs have been provided. Using the OpenTelemetry environment variables. The Airflow configs have been deprecated and will be removed in the future. [airflow.sdk._shared.observability.common] loc=common.py:87
     2026-08-19T14:50:49.460321Z [info     ] [OTLPMetricExporter] Connecting to OpenTelemetry Collector at http://grafana-k8s-monitoring-receiver-collector.observability-system.svc.cluster.local:4317 [airflow.sdk._shared.observability.common] loc=common.py:134
     >>> DagBag("/opt/airlfow/dags/gh71772.py")
     2026-08-19T14:51:22.606820Z [info     ] Filling up the DagBag from /opt/airlfow/dags/gh71772.py [airflow.dag_processing.dagbag.DagBag] loc=dagbag.py:460
     <airflow.dag_processing.dagbag.DagBag object at 0x7ba101d701a0>
     >>> DagBag("/home/karthikeyan/airflow/dags/gh71772.py").dags
     2026-08-19T14:51:34.388882Z [info     ] Filling up the DagBag from /home/karthikeyan/airflow/dags/gh71772.py [airflow.dag_processing.dagbag.DagBag] loc=dagbag.py:460
     {}
     >>> DagBag("/opt/airflow/dags/gh71772.py").dags
     2026-08-19T14:51:47.094370Z [info     ] Filling up the DagBag from /opt/airflow/dags/gh71772.py [airflow.dag_processing.dagbag.DagBag] loc=dagbag.py:460
     {'gh71772': <DAG: gh71772>}
     >>>
     ```
   </details>
   
   Means, the specific dag processor pods is able to parse the dag code in itself. So why is there the timeout? Even weirder is that the dag processor recognizes that there are python files existing, so the git fetch is also working ??!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5479019438

   @tirkarthi hmm, it sounds to me that there might be an issue with the cross-communication regarding the async subprocesses?? Otherwise, I don't see how there can come up a timeout error 🤔 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5329459224

   Question: may this behavior come from https://github.com/apache/airflow/pull/71169 or https://github.com/apache/airflow/pull/70662 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5408981700

   I checked if the Git repository for the GitDagBundle is getting fetched successfully. It does. The Git repo is successfully getting fetched, and the fetched Dag airflow_health_check can be successfully parsed by the manual approach from above with the DagBag python calss


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 or 3.3.2 [airflow]

Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-6331661701

   Update: I tested 3.3.2, same behaviour from 3.3.0 -> 3.3.2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5345088609

   Can you please set logging level as debug and also run `airflow dag-processor -vvv` to see if you get any info?
   
   ```
   [logging]
   logging_level = DEBUG
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5342548811

   The dag parses fine locally and I don't see any expensive operations at the module causing timeout. Can you please try parsing the file manually from the dag-processor pod?
   
   ```
   >>> from airflow.dag_processing.dagbag import DagBag
   >>> DagBag("/home/karthikeyan/airflow/dags/gh71772.py")
   [2026-08-19T13:05:14.830800Z] {dagbag.py:460} INFO - Filling up the DagBag from /home/karthikeyan/airflow/dags/gh71772.py
   <airflow.dag_processing.dagbag.DagBag object at 0x7f104ed73b90>
   >>> DagBag("/home/karthikeyan/airflow/dags/gh71772.py").dags
   [2026-08-19T13:05:18.127408Z] {dagbag.py:460} INFO - Filling up the DagBag from /home/karthikeyan/airflow/dags/gh71772.py
   {'gh71772': <DAG: gh71772>}
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5407642642

   @tirkarthi Short update: I enabled the debug log on both 3.3.0 and 3.3.1. Apart from the timeout message, the logs seem to be the same?? I am still trying to find something :/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5479581251

   Oh, I forgot to mention: When downgrading from Airflow 3.3.1 to Airflow 3.3.0 in the Kubernetes environment, I need to force delete the airflow-worker pods as they won't get out of "Terminating" state because the airflow-health-check DagRun is stuck in state "scheduled" I assume.
   
   I had Airflow 3.3.1 deployed for 6 days in our Sandbox cluster and the airflow-health-check DagRun was stuck in state "scheduled" this whole time, 6 days 😅 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5360258986

   Wasn't able to get to this yet, bear with me please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]

Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5407667888

   Hmm, I try a container image without our proprietary providers, maybe there is something straying poison?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org