You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "IanSweeneyKodiak (via GitHub)" <gi...@apache.org> on 2026/06/26 17:15:21 UTC
[I] dagProcessor liveness probe always fails [airflow]
IanSweeneyKodiak opened a new issue, #69048:
URL: https://github.com/apache/airflow/issues/69048
### Under which category would you file this issue?
Helm chart
### Apache Airflow version
3.2.2 (helm 1.2.2)
### What happened and how to reproduce it?
Default helm chart uses livenessProbe with `command: ~` for dagProcessor.
This uses the default command to check liveness which always fails:
`airflow jobs check --job-type DagProcessorJob --hostname $(hostname)`
Causing the dagProcessor to be restarted by k8s every 5min (livenessProbe failure)
### What you think should happen instead?
_No response_
### Operating System
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] dagProcessor liveness probe always fails [airflow]
Posted by "IanSweeneyKodiak (via GitHub)" <gi...@apache.org>.
IanSweeneyKodiak commented on issue #69048:
URL: https://github.com/apache/airflow/issues/69048#issuecomment-4811944667
Turns out there is a running dagProcessor job in the db but the cli used for the health check returns no alive jobs
`select * from job where job_type='DagProcessorJob' order by start_date desc;`
` 206 | | running | DagProcessorJob | 2026-06-26 17:33:29.530223+00 | | 2026-06-26 17:37:04.932384+00 | | airflow-dag-processor-5df45489f7-92r7z | airflow`
`airflow jobs check --job-type DagProcessorJob`
`No alive jobs found`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] dagProcessor liveness probe always fails [airflow]
Posted by "IanSweeneyKodiak (via GitHub)" <gi...@apache.org>.
IanSweeneyKodiak commented on issue #69048:
URL: https://github.com/apache/airflow/issues/69048#issuecomment-4917155627
Would it be possible/correct to update this default in the helm chart then?
`AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] dagProcessor liveness probe always fails [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on issue #69048:
URL: https://github.com/apache/airflow/issues/69048#issuecomment-4827043279
If you don't see missing dags or in the dag-processor logs, you see that the dag-processor is working, I would assume that there is high CPU usage + maybe some bottlenecks on the database, and the healthcheck command just passes the execution timeout. If that is the case, you can modify resources, maybe optimise Dags code regarding parsing and execution layers (https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code, which in general should decrease workload on the Dag Processor) and tweak a little Dag Processor configuration like parsing processor, etc.
From more fixes like workarounds, there are some options regarding the probe itself within the chart: https://github.com/apache/airflow/blob/helm-chart/1.22.0/chart/values.yaml#L3052.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] dagProcessor liveness probe always fails [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #69048:
URL: https://github.com/apache/airflow/issues/69048#issuecomment-4811764990
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] dagProcessor liveness probe always fails [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #69048:
URL: https://github.com/apache/airflow/issues/69048#issuecomment-4842359811
What is the value of `hostname` command output inside the pod? Does the value match the entries found in the database with heartbeat timestamp updated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] dagProcessor liveness probe always fails [airflow]
Posted by "arkadiuszbach (via GitHub)" <gi...@apache.org>.
arkadiuszbach commented on issue #69048:
URL: https://github.com/apache/airflow/issues/69048#issuecomment-4902179834
> Turns out there is a running dagProcessor job in the db but the cli used for the health check returns no alive jobs `select * from job where job_type='DagProcessorJob' order by start_date desc;` ` 206 | | running | DagProcessorJob | 2026-06-26 17:33:29.530223+00 | | 2026-06-26 17:37:04.932384+00 | | airflow-dag-processor-5df45489f7-92r7z | airflow`
>
> `airflow jobs check --job-type DagProcessorJob` `No alive jobs found`
>
> It seems that maybe the dagProcessor is not sending a heartbeat?
>
> Guessing this is because the heartbeat is `only_if_necessary`
>
> [airflow/airflow-core/src/airflow/jobs/dag_processor_job_runner.py](https://github.com/apache/airflow/blob/ba8cfe47e57e101a7b9ec39d4ae4a0b042042862/airflow-core/src/airflow/jobs/dag_processor_job_runner.py#L55)
>
> Line 55 in [ba8cfe4](/apache/airflow/commit/ba8cfe47e57e101a7b9ec39d4ae4a0b042042862)
>
> only_if_necessary=True,
> Maybe due to the config option `scheduler.job_heartbeat_sec: 30`
Yes its because of `scheduler.job_heartbeat_sec: 30` and `dag_processor.health_check_threshold: 30`, see: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#health-check-threshold
just increase health_check_threshold for the dag_processor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] dagProcessor liveness probe always fails [airflow]
Posted by "eajimmy (via GitHub)" <gi...@apache.org>.
eajimmy commented on issue #69048:
URL: https://github.com/apache/airflow/issues/69048#issuecomment-4916748456
Same problem here on Airflow 3.2.1 (KubernetesExecutor + CeleryExecutor).
I confirmed it with this query, running it repeatedly:
```sql
SELECT id, hostname, state,
latest_heartbeat,
EXTRACT(EPOCH FROM (now() - latest_heartbeat)) AS age_seconds
FROM job
WHERE job_type = 'DagProcessorJob'
ORDER BY latest_heartbeat DESC
LIMIT 30;
```
age_seconds oscillates continuously between ~30 and ~60s and never drops close to 0.
Setting AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120 fixes it, as mentioned above.
I also tried increasing parsing_processes and min_file_process_interval, but these had no effect on age_seconds.
It seems that the staleness is intrinsic to the heartbeat mechanism, not to parsing load.
I even confirmed this with the processor fully idle (min_file_process_interval=300, no files to parse) and the age still stayed in the 30–60s range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] dagProcessor liveness probe always fails [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on issue #69048:
URL: https://github.com/apache/airflow/issues/69048#issuecomment-4928028403
> I also tried increasing parsing_processes and min_file_process_interval, but these had no effect on age_seconds.
It seems that the staleness is intrinsic to the heartbeat mechanism, not to parsing load.
I even confirmed this with the processor fully idle (min_file_process_interval=300, no files to parse) and the age still stayed in the 30–60s range.
This indicates insufficient resources on the database, such as CPU or IOPS.
> Would it be possible/correct to update this default in the Helm chart then?
AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120
To be honest, I don't see the reason to do that. At least for now, there is a lack of information regarding particular components' configurations and resources, as well as the dags' characteristics themselves, which are impacting the load on both the dag processor and the database. From my experience with quite big Airflow environments (comparing, e.g. to the data from the Airflow Survey), when the database is correctly configured with quality gates for dags code and some tweaks to dag processor configuration (depending on the dags characteristics), the default settings are sufficient. Also, adjusting this configuration is easy, and I would say not problematic. If adjusting it wasn't possible, it would be a totally different story.
I will convert this issue to a discussion since we don't have enough data to determine whether it is environment-specific or an Apache Airflow issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] dagProcessor liveness probe always fails [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl closed issue #69048: dagProcessor liveness probe always fails
URL: https://github.com/apache/airflow/issues/69048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org