You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "n-badtke-cg (via GitHub)" <gi...@apache.org> on 2026/08/18 14:07:13 UTC
[I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
n-badtke-cg opened a new issue, #71772:
URL: https://github.com/apache/airflow/issues/71772
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
We are currently running Airflow 3.3.0 in Kubernetes using the Airflow Helm Chart v1.22 for deployment. All Dags are healthy. The Dags are fetched via Git Dag Bundle.
After deploying, the dag processor messages errors like `Processor for DagFileInfo(rel_path=PosixPath('<filepath_in_git_repo>'), bundle_name='<a_name>', bundle_path=PosixPath('/tmp/airflow/dag_bundles/<a_name>/tracking_repo', bundle_version=None) with PID NNN has been running for 50.01 seconds, exceeding the timeout of 50.00 seconds. Killing it! [airflow.dag_processing.manager.DagFileProcessorManager] loc=manager.py:1621` and the overview table now shows "# Errors" > 0
We have a very basic "airflow_health_check" Dag to verify the Airflow workers Dag runtime:
```py
from airflow import XComArg
from airflow.sdk import dag, task
from airflow.sdk.exceptions import AirflowFailException
from datetime import datetime, timedelta
HEALTH_CHECK_MESSAGE: str = "Health check successful"
HEALTHY: str = "healthy"
UNHEALTHY: str = "unhealthy"
@dag(
schedule="*/2 * * * *",
start_date=datetime(2026, 7, 1),
dagrun_timeout=timedelta(minutes=3),
is_paused_upon_creation=False,
catchup=False,
)
def airflow_health_check() -> None:
@task.bash
def health_check(text: str) -> str:
return f"echo '{text}'"
@task
def result_is_healthy(*, result: XComArg, check_against: str) -> bool:
return str(result) == check_against
@task
def print_result(is_healthy: bool) -> None:
print("is_healthy: " + str(type(is_healthy)) + " " + str(is_healthy))
print(HEALTHY if is_healthy == True else UNHEALTHY)
if not is_healthy == True:
raise AirflowFailException()
health_check_result: XComArg = health_check(HEALTH_CHECK_MESSAGE)
result_is_healthy_bool: bool = result_is_healthy(
result=health_check_result, check_against=HEALTH_CHECK_MESSAGE
)
print_result(result_is_healthy_bool)
airflow_health_check()
```
With Airflow 3.3.0, the Dag Runs are green.
Shortly after the upgrade to 3.3.1, the Dag Runs are stuck in "queued", but the Dag itself is still active.
With the Dag processor catching up and now throwing errors, the Dag gets "Deactivated".
### What you think should happen instead?
It should just work? Idk, do I overlook something?
### Operating System
Container image apache/airflow:3.3.1-python3.13
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
amazon, celery, cncf-kubernetes, common-compat, common-io, common-messaging, common-sql, databricks, docker, elasticsearch, fab, ftp, git, google, grpc, hashicorp, http, keycloak, microsoft-azure, mysql, odbc, openlineage, opensearch, postgres, redis, sendgrid, sftp, slack, smtp, snowflake, ssh, standard
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.34.0
apache-airflow-providers-celery==3.23.1
apache-airflow-providers-cncf-kubernetes==10.21.0
apache-airflow-providers-common-compat==1.18.0
apache-airflow-providers-common-io==1.8.0
apache-airflow-providers-common-messaging==2.0.4
apache-airflow-providers-common-sql==2.1.0
apache-airflow-providers-databricks==7.18.1
apache-airflow-providers-docker==4.5.9
apache-airflow-providers-elasticsearch==6.9.0
apache-airflow-providers-fab==3.8.0
apache-airflow-providers-ftp==3.15.2
apache-airflow-providers-git==0.4.2
apache-airflow-providers-google==22.3.0
apache-airflow-providers-grpc==3.9.5
apache-airflow-providers-hashicorp==4.8.0
apache-airflow-providers-http==6.0.5
apache-airflow-providers-keycloak==0.8.2
apache-airflow-providers-microsoft-azure==14.1.0
apache-airflow-providers-mysql==6.6.1
apache-airflow-providers-odbc==4.12.3
apache-airflow-providers-openlineage==2.20.0
apache-airflow-providers-opensearch==1.12.0
apache-airflow-providers-postgres==7.0.1
apache-airflow-providers-redis==4.5.0
apache-airflow-providers-sendgrid==4.2.4
apache-airflow-providers-sftp==6.0.1
apache-airflow-providers-slack==9.10.2
apache-airflow-providers-smtp==3.0.3
apache-airflow-providers-snowflake==6.16.0
apache-airflow-providers-ssh==6.0.1
apache-airflow-providers-standard==1.17.0
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
v1.35.1
### Helm Chart configuration
Incomplete, only an excerpt. Please let me know if there might be something other relevant to share.
```yaml
airflowVersion: 3.3.1
config:
azure_remote_logging:
remote_wasb_log_container: airflow-remote-logs
celery_broker_transport_options:
visibility_timeout: 14400
core:
auth_manager: airflow.providers.keycloak.auth_manager.keycloak_auth_manager.KeycloakAuthManager
test_connection: Hidden
keycloak_auth_manager:
realm: application
requests_pool_size: 20
server_url: <a_valid_url>
logging:
delete_local_logs: "False"
remote_base_log_folder: wasb://logs
remote_log_conn_id: airflow-remote-logging-wasb
remote_logging: "True"
dagProcessor:
dagBundleConfigList:
- classpath: airflow.providers.git.bundles.git.GitDagBundle
kwargs:
git_conn_id: <a_name_for_conn>
refresh_interval: 30
tracking_ref: main
name: <a_name>
```
### Docker Image customizations
Dockerfile:
```dockerfile
ARG AIRFLOW_CONTAINER_VERSION=3.3.0-python3.13
FROM apache/airflow:${AIRFLOW_CONTAINER_VERSION}
USER root
COPY org_ca.pem /tmp/org_ca.pem
RUN csplit -z -f /usr/local/share/ca-certificates/org_ca_ \
--suffix-format='%03d.crt' \
/tmp/org_ca.pem \
'/-----BEGIN CERTIFICATE-----/' '{*}' \
&& update-ca-certificates \
&& rm /tmp/org_ca.pem
USER airflow
COPY upstream-requirements.txt upstream-requirements.txt
RUN --mount=type=secret,id=pip-conf,dst=/etc/pip.conf,mode=0440 \
pip install --no-cache-dir \
--constraint /home/airflow/constraints.txt \
--requirement upstream-requirements.txt
# install custom providers and packages in separate pip install command to avoid dependency version clash with constraints
COPY custom-requirements.txt custom-requirements.txt
RUN --mount=type=secret,id=pip-conf,dst=/etc/pip.conf,mode=0440 \
pip install --no-cache-dir \
--requirement custom-requirements.txt
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
```
upstream-requirements.txt:
```
# Environmernt variable ${AIRFLOW_VERSION} comes from the upstream Airflow image and is resolved by pip
apache-airflow[cloudpickle,databricks,keycloak,microsoft.azure,otel]==${AIRFLOW_VERSION}
# Airflow components not covered via extras, versions are covered by constraints file
virtualenv
```
custom-requirements.txt:
```
# Environment variable ${AIRFLOW_VERSION} comes from the upstream Airflow image and is resolved by pip
# Pin Airflow version to prevent pip from changing it during dependency resolution
apache-airflow==${AIRFLOW_VERSION}
<some propietary providers>
```
### Anything else?
To be honest, I am a bit lost where to look at else. Please let me know :)
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5343978725
@tirkarthi Given your commands, I opened a remote shell to the Kubernetes Pod did what you asked for:
<details>
<summary>bash commands dump (Click to expand)</summary>
```bash
$ kubectl exec -n airflow airflow-dag-processor-66894c6b74-bkp4d -c dag-processor -it -- bash -il
airflow@airflow-dag-processor-66894c6b74-bkp4d:/opt/airflow$ cd dags
airflow@airflow-dag-processor-66894c6b74-bkp4d:/opt/airflow/dags$ cat > gh71772.py<< EOF
> from __future__ import annotations
from datetime import datetime, timedelta
from airflow import XComArg
from airflow.sdk import dag, task
from airflow.sdk.exceptions import AirflowFailException
HEALTH_CHECK_MESSAGE: str = "Health check successful"
HEALTHY: str = "healthy"
UNHEALTHY: str = "unhealthy"
@dag(
schedule="*/2 * * * *",
start_date=datetime(2026, 7, 1),
dagrun_timeout=timedelta(minutes=3),
is_paused_upon_creation=False,
catchup=False,
)
def gh71772() -> None:
@task.bash
def health_check(text: str) -> str:
return f"echo '{text}'"
@task
def result_is_healthy(*, result: XComArg, check_against: str) -> bool:
return str(result) == check_against
@task
def print_result(is_healthy: bool) -> None:
print("is_healthy: " + str(type(is_healthy)) + " " + str(is_healthy))
print(HEALTHY if is_healthy == True else UNHEALTHY)
if not is_healthy == True:
raise AirflowFailException()
health_check_result: XComArg = health_check(HEALTH_CHECK_MESSAGE)
result_is_healthy_bool: bool = result_is_healthy(
result=health_check_result, check_against=HEALTH_CHECK_MESSAGE
)
print_result(result_is_healthy_bool)
gh71772()
> EOF
airflow@airflow-dag-processor-66894c6b74-bkp4d:/opt/airflow/dags$ cat gh71772.py
from __future__ import annotations
from datetime import datetime, timedelta
from airflow import XComArg
from airflow.sdk import dag, task
from airflow.sdk.exceptions import AirflowFailException
HEALTH_CHECK_MESSAGE: str = "Health check successful"
HEALTHY: str = "healthy"
UNHEALTHY: str = "unhealthy"
@dag(
schedule="*/2 * * * *",
start_date=datetime(2026, 7, 1),
dagrun_timeout=timedelta(minutes=3),
is_paused_upon_creation=False,
catchup=False,
)
def gh71772() -> None:
@task.bash
def health_check(text: str) -> str:
return f"echo '{text}'"
@task
def result_is_healthy(*, result: XComArg, check_against: str) -> bool:
return str(result) == check_against
@task
def print_result(is_healthy: bool) -> None:
print("is_healthy: " + str(type(is_healthy)) + " " + str(is_healthy))
print(HEALTHY if is_healthy == True else UNHEALTHY)
if not is_healthy == True:
raise AirflowFailException()
health_check_result: XComArg = health_check(HEALTH_CHECK_MESSAGE)
result_is_healthy_bool: bool = result_is_healthy(
result=health_check_result, check_against=HEALTH_CHECK_MESSAGE
)
print_result(result_is_healthy_bool)
gh71772()
airflow@airflow-dag-processor-66894c6b74-bkp4d:/opt/airflow/dags$ python
Python 3.13.14 (main, Aug 12 2026, 08:57:09) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from airflow.dag_processing.dagbag import DagBag
...
KeyboardInterrupt
>>> from airflow.dag_processing.dagbag import DagBag
2026-08-19T14:50:49.459701Z [warning ] Both the standard OpenTelemetry environment variables and the Airflow OpenTelemetry configs have been provided. Using the OpenTelemetry environment variables. The Airflow configs have been deprecated and will be removed in the future. [airflow.sdk._shared.observability.common] loc=common.py:87
2026-08-19T14:50:49.460321Z [info ] [OTLPMetricExporter] Connecting to OpenTelemetry Collector at http://grafana-k8s-monitoring-receiver-collector.observability-system.svc.cluster.local:4317 [airflow.sdk._shared.observability.common] loc=common.py:134
>>> DagBag("/opt/airlfow/dags/gh71772.py")
2026-08-19T14:51:22.606820Z [info ] Filling up the DagBag from /opt/airlfow/dags/gh71772.py [airflow.dag_processing.dagbag.DagBag] loc=dagbag.py:460
<airflow.dag_processing.dagbag.DagBag object at 0x7ba101d701a0>
>>> DagBag("/home/karthikeyan/airflow/dags/gh71772.py").dags
2026-08-19T14:51:34.388882Z [info ] Filling up the DagBag from /home/karthikeyan/airflow/dags/gh71772.py [airflow.dag_processing.dagbag.DagBag] loc=dagbag.py:460
{}
>>> DagBag("/opt/airflow/dags/gh71772.py").dags
2026-08-19T14:51:47.094370Z [info ] Filling up the DagBag from /opt/airflow/dags/gh71772.py [airflow.dag_processing.dagbag.DagBag] loc=dagbag.py:460
{'gh71772': <DAG: gh71772>}
>>>
```
</details>
Means, the specific dag processor pods is able to parse the dag code in itself. So why is there the timeout? Even weirder is that the dag processor recognizes that there are python files existing, so the git fetch is also working ??!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5479019438
@tirkarthi hmm, it sounds to me that there might be an issue with the cross-communication regarding the async subprocesses?? Otherwise, I don't see how there can come up a timeout error 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5329459224
Question: may this behavior come from https://github.com/apache/airflow/pull/71169 or https://github.com/apache/airflow/pull/70662 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5408981700
I checked if the Git repository for the GitDagBundle is getting fetched successfully. It does. The Git repo is successfully getting fetched, and the fetched Dag airflow_health_check can be successfully parsed by the manual approach from above with the DagBag python calss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 or 3.3.2 [airflow]
Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-6331661701
Update: I tested 3.3.2, same behaviour from 3.3.0 -> 3.3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5345088609
Can you please set logging level as debug and also run `airflow dag-processor -vvv` to see if you get any info?
```
[logging]
logging_level = DEBUG
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5342548811
The dag parses fine locally and I don't see any expensive operations at the module causing timeout. Can you please try parsing the file manually from the dag-processor pod?
```
>>> from airflow.dag_processing.dagbag import DagBag
>>> DagBag("/home/karthikeyan/airflow/dags/gh71772.py")
[2026-08-19T13:05:14.830800Z] {dagbag.py:460} INFO - Filling up the DagBag from /home/karthikeyan/airflow/dags/gh71772.py
<airflow.dag_processing.dagbag.DagBag object at 0x7f104ed73b90>
>>> DagBag("/home/karthikeyan/airflow/dags/gh71772.py").dags
[2026-08-19T13:05:18.127408Z] {dagbag.py:460} INFO - Filling up the DagBag from /home/karthikeyan/airflow/dags/gh71772.py
{'gh71772': <DAG: gh71772>}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5407642642
@tirkarthi Short update: I enabled the debug log on both 3.3.0 and 3.3.1. Apart from the timeout message, the logs seem to be the same?? I am still trying to find something :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5479581251
Oh, I forgot to mention: When downgrading from Airflow 3.3.1 to Airflow 3.3.0 in the Kubernetes environment, I need to force delete the airflow-worker pods as they won't get out of "Terminating" state because the airflow-health-check DagRun is stuck in state "scheduled" I assume.
I had Airflow 3.3.1 deployed for 6 days in our Sandbox cluster and the airflow-health-check DagRun was stuck in state "scheduled" this whole time, 6 days 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5360258986
Wasn't able to get to this yet, bear with me please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues upgrading from Airflow 3.3.0 to 3.3.1 [airflow]
Posted by "n-badtke-cg (via GitHub)" <gi...@apache.org>.
n-badtke-cg commented on issue #71772:
URL: https://github.com/apache/airflow/issues/71772#issuecomment-5407667888
Hmm, I try a container image without our proprietary providers, maybe there is something straying poison?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org