You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "anon-airflow (via GitHub)" <gi...@apache.org> on 2024/11/18 08:25:36 UTC

[I] Scheduler stuck when "too old resource version" error is thrown [airflow]

anon-airflow opened a new issue, #44127:
URL: https://github.com/apache/airflow/issues/44127

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.8.2
   
   ### What happened?
   
   I get the following error a lot on my airflow scheduler pods:
   
   ```
   {kubernetes_executor_utils.py:121} ERROR - Unknown error in KubernetesJobWatcher. Failing
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py", line 112, in run
       self.resource_version = self._run(
                               ^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py", line 168, in _run
       for event in self._pod_events(kube_client=kube_client, query_kwargs=kwargs):
     File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/watch/watch.py", line 182, in stream
       raise client.rest.ApiException(
   kubernetes.client.exceptions.ApiException: (410)
   Reason: Expired: too old resource version: 725263658 (725300129)
   
   Process KubernetesJobWatcher-8:
   Traceback (most recent call last):
     File "/usr/local/lib/python3.8/multiprocessing/process.py", line 314, in _bootstrap
       self.run()
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py", line 112, in run
       self.resource_version = self._run(
                               ^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py", line 168, in _run
       for event in self._pod_events(kube_client=kube_client, query_kwargs=kwargs):
     File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/watch/watch.py", line 182, in stream
       raise client.rest.ApiException(
   kubernetes.client.exceptions.ApiException: (410)
   Reason: Expired: too old resource version: 725263658 (725300129)
   ```
   
   When this error appears relatively many times on my airflow scheduler pods, All my DAG runs become very slow- This is expressed in the fact that the amount of my "scheduled" slots is very high and in contrast the amount of my "'queued" and "running" slots is very low (about 15 slots together) even though I have defined 128 slots.
   
   Also my resource utilization in my namespace is very low (20% cpu and memory usage) so the problem is not resources either.
   
   NOTE: I use the package "apache-airflow-providers-cncf-kubernetes" on version 8.0.0 as required for Airflow 2.8.2 according to the constraints.
   
   ### What you think should happen instead?
   
   I think Airflow should know how to handle this error so that even when the error is thrown, the scheduler should continue to work  properly and not "freeze".
   
   ### How to reproduce
   
   I think it would happen on any deployment in this version of Airflow with running DAGs. 
   
   ### Operating System
   
   rhel 8
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-cncf-kubernetes==8.0.0
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   We are in a private cloud with constraints, we took the most of the chart but handled the constraints ourselves.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler stuck when "too old resource version" error is thrown [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #44127:
URL: https://github.com/apache/airflow/issues/44127#issuecomment-2578930352

   This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler stuck when "too old resource version" error is thrown [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #44127:
URL: https://github.com/apache/airflow/issues/44127#issuecomment-2482254229

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler stuck when "too old resource version" error is thrown [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #44127:
URL: https://github.com/apache/airflow/issues/44127#issuecomment-2505926743

   I think there were some recent changes in Kubernetes Provider to address similar issues: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/changelog.html -> look it up and best - upgrade Airflow to the latest version (or only K8S provider)
   
   https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-and-upgrade-scenarios
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler stuck when "too old resource version" error is thrown [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #44127:
URL: https://github.com/apache/airflow/issues/44127#issuecomment-2625997896

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler stuck when "too old resource version" error is thrown [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #44127:
URL: https://github.com/apache/airflow/issues/44127#issuecomment-2540264837

   This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler stuck when "too old resource version" error is thrown [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #44127:
URL: https://github.com/apache/airflow/issues/44127#issuecomment-2505932737

   Please come-back to us with results after upgrade .
   
   BTW.
   
   > NOTE: I use the package "apache-airflow-providers-cncf-kubernetes" on version 8.0.0 as required for Airflow 2.8.2 according to the constraints.
   
   No. You understand it backwards. Constraints are NOT requirements - you are free to upgrade stuff after initial installation. Constraints are there EXCLUSIVELY to give you reproducible installation and non-conflicting requirements - 
   see detailed description why we need constraints and how you are supposed to use them https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#constraints-files
   
   Also you can see my talk from Airlfow Summit 2023 https://www.youtube.com/watch?v=zPjIQjjjyHI describing how to manage dependencies for Airlfow.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler stuck when "too old resource version" error is thrown [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #44127: Scheduler stuck when "too old resource version" error is thrown
URL: https://github.com/apache/airflow/issues/44127


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler stuck when "too old resource version" error is thrown [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #44127:
URL: https://github.com/apache/airflow/issues/44127#issuecomment-2559864317

   Possibly then somoene who follows k8s part can help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler stuck when "too old resource version" error is thrown [airflow]

Posted by "anon-airflow (via GitHub)" <gi...@apache.org>.
anon-airflow commented on issue #44127:
URL: https://github.com/apache/airflow/issues/44127#issuecomment-2543465797

   I upgrade my airflow environment to use in the latest version of "apache-airflow-provider-cncf-kubernetes" package (10.0.1) and the problem didn't solved - the error is still thrown.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org