You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "alex-magno (via GitHub)" <gi...@apache.org> on 2023/09/01 14:29:35 UTC

[GitHub] [airflow] alex-magno opened a new issue, #34013: Task is not able to be run - k8s Celery workers with stale resources

alex-magno opened a new issue, #34013:
URL: https://github.com/apache/airflow/issues/34013

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   We are facing a quite interesting problem with our celery workers. **We run on AWS EKS (m-series nodes running Bottlerocket OS), Airflow v2.6.2**.
   
   When the amount of running tasks is high (close to our max cap), we notice an increase in the celery workers memory usage followed by a stale period. At the same time, CPU utilization goes to zero.
   ![image](https://github.com/apache/airflow/assets/87935605/55b8faba-0b46-48f5-a22b-a4a0b98713d6)
   
   During this time, the worker container seems to freeze (no logs are produced). Of the tasks that were running, almost all of them failed with the only log being:
   ```
   [2023-08-31, 13:20:20 UTC] {taskinstance.py:1093} INFO - Dependencies not met for <TaskInstance: ***** scheduled__2023-08-31T10:00:00+00:00 [failed]>, dependency 'Task Instance State' FAILED: Task is in the 'failed' state.
   [2023-08-31, 13:20:20 UTC] {local_task_job_runner.py:154} INFO - Task is not able to be run
   ```
   
   It is important to note that this problem happens sometimes, but not every time. Which makes it even more confusing. Also, workers pods are not killed in the process, nor do they throw any exceptions.
   
   Some of our configurations are:
   ```
   AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG = 32
   AIRFLOW__CORE__PARALLELISM = 64
   AIRFLOW__CELERY__WORKER_CONCURRENCY = 12
   ```
   
   We are running at the moment with 5 celery workers, with no CPU or memory limits. The scheduler seems fine during those periods, producing regular logs. The only thing I can think of at the moment is that its probably due to a shortage of resources in the celery workers? However the worker concurrency limit should prevent a high usage of CPU/memory. Any ideas will be appreciated.
   
   Some executor metrics on the same time period:
   ![image](https://github.com/apache/airflow/assets/87935605/64157a01-dfad-4c0e-ad26-e0a005270f7f)
   ![image](https://github.com/apache/airflow/assets/87935605/dec42938-1285-4fa1-88a7-e1d64293ac86)
   
   ### What you think should happen instead
   
   CPU and Memory resources should not go stale. Also, workers should keep logging as usual, instead of "freezing".
   
   ### How to reproduce
   
   Deploy through Helm chart, using the `CeleryExecutor`. Run the maximum number of allowed tasks at the same time. However, the problem is not guaranteed to happen.
   
   ### Operating System
   
   Bottlerocket OS
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   AWS EKS with k8s `v1.26.1` - Nodes (m-series) using Bottlerocket OS.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "keysersoza (via GitHub)" <gi...@apache.org>.
keysersoza commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2150271955

   We could mitigate the issue a bit by setting these configs:
   ```
     celery:
       task_publish_max_retries: 5
     celery_broker_transport_options:
       visibility_timeout: 86400
   ```
   
   The issues started being less frequent, but Tasks still get killed sometimes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2016216787

   >Can everyone in this thread make an experiment, disable it and report back here - at least this will give some clue on where the problems might happen.
   
   Yes please, @alex-magno Could you try with disabling mini-scheduler?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-1850303636

   > I have been seeing this error appearing after upgrading to 2.7.3, also using the Celery Executor on K8s. However, I don't see that pattern when looking at resource consumption. I wonder what "Dependencies not met ..." means
   
   I think we miss some details but if you can provide mroe @babaMar, logs, explanation what you do, circusmstances where it happens, it might make it easier for someone to diagnose it. Without it and with cryptic " Dependencies not met .." comment it neither helps nor explains what your problem is. And it might be different propblem, so what I suggest is to open a discussion where you will explain everything you see and experience. The more detailed it is, the higher chance that someeone will decide to spend their time on it and maybe even diagnose it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #34013: Task is not able to be run - k8s Celery workers with stale resources

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-1704411889

   For me it looks like something indeed takes a lot of memory for the workers but what it would be - hard to say.
   Is it possible to run your workers with memray (https://github.com/bloomberg/memray) ? It would be great if it could produce flamechart with peak memory usage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-1851859638

   Well  I think the crypticity is that you posted it in completely wrong thread. Your comment and log have nothing to do with the problem here (or so it seems). . I suggest you open a discussion where you explain more information about your DAGs 
   
   The more details you provide , the higher chance us that someone will help you. Note that this is forum where people help others in their free time - when they see they can help. So the more focused and detailed your description is, the higher chance is that someone will actually spend their time helping (remember it's free time of people here whole you get the software for absolutely free without any guarantees).
   
   So it's up to you if you want to get help how detailed your own analysis will be to help those who might want to help you.
   
   And yes if you get helped then next thing is we will ask you to make contribution to make the message less cryptic. Like 2700 other contributors.
   
   Just a friendly advice as you might not realize that open source means community 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "keysersoza (via GitHub)" <gi...@apache.org>.
keysersoza commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2148580333

   Hi @pavelpi , we are experiencing the same issue: https://github.com/apache/airflow/issues/40054
   
   We could not observe any resource related issue; in our case, tasks go almost immediately in `up for retry` state and then fail.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #34013: Task is not able to be run - k8s Celery workers with stale resources
URL: https://github.com/apache/airflow/issues/34013


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "babaMar (via GitHub)" <gi...@apache.org>.
babaMar commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-1851596900

   > > I have been seeing this error appearing after upgrading to 2.7.3, also using the Celery Executor on K8s. However, I don't see that pattern when looking at resource consumption. I wonder what "Dependencies not met ..." means
   > 
   > I think we miss some details but if you can provide mroe @babaMar, logs, explanation what you do, circusmstances where it happens, it might make it easier for someone to diagnose it. Without it and with cryptic " Dependencies not met .." comment it neither helps nor explains what your problem is. And it might be different propblem, so what I suggest is to open a discussion where you will explain everything you see and experience. The more detailed it is, the higher chance that someeone will decide to spend their time on it and maybe even diagnose it.
   
   @potiuk the "cripticity" is already in the lack of logs in Airflow:
   ```
   [2023-12-12, 08:03:01 UTC] {taskinstance.py:1149} INFO - Dependencies not met for <TaskInstance: ***__2023-12-11T06:20:00+00:00 [failed]>, dependency 'Task Instance State' FAILED: Task is in the 'failed' state.
   [2023-12-12, 08:03:01 UTC] {local_task_job_runner.py:154} INFO - Task is not able to be run
   ```
   The circumstances are not special, just sporadic failures with the logs above. The problem is indeed that I have no details whatsoever. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "suvorom (via GitHub)" <gi...@apache.org>.
suvorom commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-3032004702

   Facing the same behaviour with Kubernetes Executor (AWS EKS, karpenter) in Airflow 2.5.2
   ```
   [2025-07-03, 07:31:50 UTC] {taskinstance.py:1074} INFO - Dependencies not met for <TaskInstance: abc-daily-01 scheduled__2025-07-01T06:00:00+00:00 [scheduled]>, dependency 'Task Instance State' FAILED: Task is in the 'scheduled' state.
   [2025-07-03, 07:31:50 UTC] {local_task_job.py:151} INFO - Task is not able to be run
   ```
   Did anyone find a solution?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "elmoutassim-mahdaoui-ad (via GitHub)" <gi...@apache.org>.
elmoutassim-mahdaoui-ad commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2012298111

   Also facing this issue on my side. Any solution yet?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "alokshenoy (via GitHub)" <gi...@apache.org>.
alokshenoy commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2453251360

   We're seeing a similar issue. Will attempt disabling the mini-scheduler and see if that makes a difference and revert back.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "keysersoza (via GitHub)" <gi...@apache.org>.
keysersoza commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2148596956

   @potiuk I tried to disable the mini scheduler, but it did not fix the issue. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "soravispr (via GitHub)" <gi...@apache.org>.
soravispr commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-1940396284

   Also facing this issue. Any solution yet?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #34013: Task is not able to be run - k8s Celery workers with stale resources

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-1702840251

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2839607649

   @alokshenoy Does setting following help like it helped @keysersoza :
   
   ```
     celery:
       task_publish_max_retries: 5
     celery_broker_transport_options:
       visibility_timeout: 86400
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2015639112

   No. I do not see anyone working on it nor providing a solution.  But the log above suggests that this is caused by so called "mini scheduler" which is enabled by this configuration https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#schedule-after-task-execution - likely disabling it will mitigate the problem (at the expense of slightly increased latency between tasks.
   
   Can everyone in this thread make an experiment, disable it and report back here - at least this will give some clue on where the problems might happen.
   
   cc: @kaxil @ephraimbuddy -> problems similar to that one keep on appearing, I think there is some kind of deadlock/resource contention with mini scheduler in some specific cases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "babaMar (via GitHub)" <gi...@apache.org>.
babaMar commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2069604724

   @pavelpi what I learned so far is that tasks in this state get killed by external processes, typically of the OS. For example, I found a DAG that would run 5 parallel tasks and each task would load a dataset into disk. It turned out, it would run out of disk space and the OS would just kill those processes of course.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "babaMar (via GitHub)" <gi...@apache.org>.
babaMar commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-1849565102

   I have been seeing this error appearing after upgrading to 2.7.3, also using the Celery Executor on K8s. However, I don't see that pattern when looking at resource consumption. I wonder what "Dependencies not met ..." means 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "pavelpi (via GitHub)" <gi...@apache.org>.
pavelpi commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2069583249

   Fails also on our side. Airflow 2.7.3.
   We have 5 tasks running in parallel which all succeed. The task that is run then reports `dependency 'Task Instance State' FAILED: Task is in the 'failed' state.`
   This happens in about 15% of runs of given DAG


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "anderzhao (via GitHub)" <gi...@apache.org>.
anderzhao commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-2014304310

   Also facing this issue. Any solution yet? 
   AIRFLOW_VERSION=2.7.2
   ```python
   file_sensor = FileSensor(
       task_id=f"file_sensor_***",
       filepath="***",
       poke_interval=timedelta(minutes=10),
       mode='reschedule',
       timeout=20*60*60,
   )
   ```
   
   ```bash
   [2024-03-22T00:20:22.482+0800] {local_task_job_runner.py:228} INFO - Task exited with return code 0
   [2024-03-22T00:20:23.018+0800] {taskinstance.py:2778} INFO - 0 downstream tasks scheduled from follow-on schedule check
   ...
   [2024-03-22T00:30:33.300+0800] {local_task_job_runner.py:228} INFO - Task exited with return code 0
   [2024-03-22T00:30:33.350+0800] {taskinstance.py:2778} INFO - 0 downstream tasks scheduled from follow-on schedule check
   ...
   [2024-03-22T00:47:46.183+0800] {local_task_job_runner.py:228} INFO - Task exited with return code 0
   [2024-03-22T00:47:46.220+0800] {taskinstance.py:2778} INFO - 0 downstream tasks scheduled from follow-on schedule check
   [2024-03-22T01:13:34.393+0800] {taskinstance.py:1149} INFO - Dependencies not met for <TaskInstance: dag_name.task_name scheduled__2024-03-20T16:00:00+00:00 [failed]>, dependency 'Task Instance State' FAILED: Task is in the 'failed' state.
   [2024-03-22T01:13:34.401+0800] {local_task_job_runner.py:154} INFO - Task is not able to be run
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-5756778260

   This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #34013:
URL: https://github.com/apache/airflow/issues/34013#issuecomment-6334956355

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org