You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/23 11:53:25 UTC

[GitHub] [airflow] ephraimbuddy opened a new pull request #19769: Handle stuck queued tasks in Celery

ephraimbuddy opened a new pull request #19769:
URL: https://github.com/apache/airflow/pull/19769


   Move the state of stuck queued tasks in Celery to Scheduled so that
   the Scheduler can queue them again
   Closes: https://github.com/apache/airflow/issues/19699
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1029716469


   And finally (hopefully) I noticed there's been a significant performance degradation in our whole pipeline this morning as it was the first time we ran this patch in production.
   
   It's still processing and no stuck tasks in "Queued", so that is very positive, but usually the pipeline is done since quite a while back, but right now it's still in the middle of processing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1029801712


   cc: @ash @jedcunningham 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1062651719


   @Chris7 can you help and test the 3 patches here?
   1: Patch #19769
   2: Patch #21335
   3: Patch #21556
   
   Otherwise, we may have to revert this change


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1031695292


   @jedcunningham Gladly, I'll see if I can deploy it tomorrow and how it behaves.
   Just to clarify, this would be patching the system with both 19769 and 21335?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r783205515



##########
File path: airflow/config_templates/config.yml
##########
@@ -1678,6 +1678,13 @@
       type: string
       example: ~
       default: "False"
+    - name: stuck_queued_task_check_interval
+      description: |
+        How often to check for stuck queued task (in seconds)
+      version_added: 2.2.3

Review comment:
       ```suggestion
         version_added: 2.3.0
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1033838250


   That means that the PR didn't fix the issue then cc: @jedcunningham 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1033461316


   @ephraimbuddy and rest of the Airflow team monitoring this ticket.
   I'm about to test the two patches and will get back with results later today.
   
   But I also need to update this ticket with some other information as well.
   Last week we deployed the patch for 19769 and on Friday and Saturday morning the system/pipeline was incredibly delayed due to the scheduler running at a fraction of the normal speed. We increased the CPU allocation for the scheduler as mentioned and the pipeline finished in a better shape, but it still wasn't as fast as a "clean" Airflow 2.2.3 installation.
   I had also changed the settings to 1800 & 3600 seconds for clearing the Celery tasks as opposed to the standard 300 seconds.
   
   I left the system running with the patch in place and increased CPU allocation, and when I checked in on the system on Sunday morning all was calm... too calm it turned out as nothing was being scheduled by the scheduler. DAGs would go "green" in running, but no tasks whatsoever started. Of course this meant no errors since nothing ran. 
   
   In the end I had to redeploy our system back to a clean 2.2.3 installation and then the scheduler started working again. Of course with a few tasks that ended up in "Queued" state, but compared to the other mornings, nothing major.
   
   The next part is only speculation, but the scheduler felt like it got slower and slower the longer it ran. With the increased time in between runs it took longer for the problem to surface, but like I said, on Sunday morning the scheduler was ground to a halt. 
   Note also that the Airflow webserver **didn't** complain about the scheduler not being online, it could apparently communicate with the scheduler just fine. It was just that the scheduler didn't start any new tasks.
   
   Again I'll test the two patches today in our staging environment and get back with the results. I just thought it was important to get the information from this weekend to you as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] SamWheating commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
SamWheating commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r783312574



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +385,49 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session: Session = NEW_SESSION) -> None:
+        """
+        Tasks can get stuck in queued state in DB while still not in
+        worker. This happens when the worker is autoscaled down and
+        the task is queued but has not been picked up by any worker prior to the scaling.
+
+        In such situation, we update the task instance state to scheduled so that
+        it can be queued again. We chose to use task_adoption_timeout to decide

Review comment:
       ```suggestion
           it can be queued again. We chose to use task_adoption_timeout to decide when
           a queued task is considered stuck and should be reschelduled.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] Chris7 commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
Chris7 commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1069179669


   Thanks for the work on this @ephraimbuddy. I am not able to test right now. Our previous usage pattern was farming out a ton of celery tasks that were monitoring work being done by a remote service (Batch). We've since moved these to use the deferred pattern to monitor these tasks (this was a great thing to gain in the most recent upgrade).
   
   So while I think the scenario still exists from my bug report, the chance of us hitting it are drastically lower (and we haven't hit it again yet) as the majority of our tasks are now in deferred. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r755115989



##########
File path: airflow/executors/celery_executor.py
##########
@@ -337,6 +344,12 @@ def sync(self) -> None:
 
         if self.adopted_task_timeouts:
             self._check_for_stalled_adopted_tasks()
+        # Use EventScheduler to avoid checking this too often
+        self.event_scheduler = EventScheduler()
+        self.event_scheduler.call_regular_interval(
+            self.stuck_queued_task_check_interval,
+            self._clear_stuck_queued_tasks(),
+        )

Review comment:
       I will create the timer at `start` method. Thanks!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r756613516



##########
File path: tests/executors/test_celery_executor.py
##########
@@ -411,6 +404,107 @@ def test_check_for_stalled_adopted_tasks_goes_in_ordered_fashion(self):
         assert executor.running == {key_2}
         assert executor.adopted_task_timeouts == {key_2: queued_dttm_2 + executor.task_adoption_timeout}
 
+    @pytest.mark.backend("mysql", "postgres")
+    @pytest.mark.parametrize(
+        "state, queued_dttm, executor_id",
+        [
+            (State.SCHEDULED, timezone.utcnow() - timedelta(days=2), '231'),
+            (State.QUEUED, timezone.utcnow(), '231'),
+            (State.QUEUED, timezone.utcnow(), None),
+        ],
+    )
+    def test_stuck_queued_tasks_are_cleared(
+        self, state, queued_dttm, executor_id, session, dag_maker, create_dummy_dag, create_task_instance
+    ):
+        """Test that clear_stuck_queued_tasks works"""
+        ti = create_task_instance(state=State.QUEUED)
+        ti.queued_dttm = queued_dttm
+        ti.external_executor_id = executor_id
+        session.merge(ti)
+        session.flush()
+        executor = celery_executor.CeleryExecutor()
+        executor._clear_stuck_queued_tasks()
+        session.flush()
+        ti = session.query(TaskInstance).filter(TaskInstance.task_id == ti.task_id).one()
+        assert ti.state == state
+
+    @pytest.mark.backend("mysql", "postgres")
+    def test_task_in_queued_tasks_dict_are_not_cleared(
+        self, session, dag_maker, create_dummy_dag, create_task_instance
+    ):
+        """Test that clear_stuck_queued_tasks doesn't clear tasks in executor.queued_tasks"""
+        ti = create_task_instance(state=State.QUEUED)
+        ti.queued_dttm = timezone.utcnow() - timedelta(days=2)
+        ti.external_executor_id = '231'
+        session.merge(ti)
+        session.flush()
+        executor = celery_executor.CeleryExecutor()
+        executor.queued_tasks = {ti.key: AsyncResult("231")}
+        executor._clear_stuck_queued_tasks()
+        session.flush()
+        ti = session.query(TaskInstance).filter(TaskInstance.task_id == ti.task_id).one()
+        assert ti.state == State.QUEUED
+
+    @pytest.mark.backend("mysql", "postgres")
+    def test_task_in_running_dict_are_not_cleared(
+        self, session, dag_maker, create_dummy_dag, create_task_instance
+    ):
+        """Test that clear_stuck_queued_tasks doesn't clear tasks in executor.running"""
+        ti = create_task_instance(state=State.QUEUED)
+        ti.queued_dttm = timezone.utcnow() - timedelta(days=2)
+        ti.external_executor_id = '231'
+        session.merge(ti)
+        session.flush()
+        executor = celery_executor.CeleryExecutor()
+        executor.running = {ti.key: AsyncResult("231")}
+        executor._clear_stuck_queued_tasks()
+        session.flush()
+        ti = session.query(TaskInstance).filter(TaskInstance.task_id == ti.task_id).one()
+        assert ti.state == State.QUEUED
+
+    @pytest.mark.backend("mysql", "postgres")
+    @mock.patch("airflow.executors.celery_executor.AsyncResult")
+    def test_task_in_not_pending_in_asyncresults_are_not_cleared(
+        self, asyncresult_mock, session, dag_maker, create_dummy_dag, create_task_instance
+    ):
+        """Test that task in other state aside pending in AsyncResult are not cleared"""
+        ti = create_task_instance(state=State.QUEUED)
+        ti.queued_dttm = timezone.utcnow() - timedelta(days=2)
+        ti.external_executor_id = '231'
+        session.merge(ti)
+        session.flush()
+        executor = celery_executor.CeleryExecutor()
+        executor._clear_stuck_queued_tasks()
+        session.flush()
+        ti = session.query(TaskInstance).filter(TaskInstance.task_id == ti.task_id).one()
+        # Since AsyncResult is mocked, the returned_value is not PENDING
+        # We end up not clearing the task
+        assert ti.state == State.QUEUED
+
+    @pytest.mark.backend("mysql", "postgres")
+    @pytest.mark.parametrize(
+        "last_check_time, state",
+        [
+            (time.time() - 400, State.SCHEDULED),
+            (time.time() - 200, State.QUEUED),

Review comment:
       Consider using `freezegun` here—calculating this against actual time may cause issues if the test suite is running slowly, since these `time.time()` are evaluated at import time, but the test may not actually start immediately after import (if the test runner decides to run other tests first).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1029755436


   Quick update on this: After I live-patched our GCP/Kubernetes cluster and the scheduler specifically to use more CPU the situation is better. We still see the warning that the scheduler does not appear to be running, just a lot less.
   
   Normally our airflow-scheduler (release 2.2.2) pod runs on "1.5" CPU in the kubernetes cluster, and increasing it to "2" CPU last night didn't help a lot. But when I increased it to "3" this morning the situation got a lot better and the scheduler was running a bit smoother when I followed both it's log and also looking at the DAGs/tasks in the webserver. 
   
   Good news though, no stuck tasks in "Queued" and there was a lot of ramp-up and down on workers that normally would have occurred in this case.
   
   So this patch seems to have introduced a higher CPU load/requirement on the scheduler (In our context 1.5CPU => 3CPU).
   Below is the CPU graph from our morning run and when I increased the resources to 3CPU. As you can see it still hits the limit, but overall has some more breathing room, confirmed by the systems behavior.
   
   ![image](https://user-images.githubusercontent.com/2227006/152496064-e97d24de-28b1-49ea-95cc-f745c0e46d41.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1025823384


   @ephraimbuddy Thanks :+1: 
   I already removed all the test_* files (not anything else right?), thanks for the help on the NEW_SESSION.
   Is NEW_SESSION part of 2.2.4 and 2.3.0 then? 
   
   I'll update here with a comment if you want to after I tested the patch :+1: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1006444793


   There are some sqlite tests failing here - likely related.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r782897587



##########
File path: tests/executors/test_celery_executor.py
##########
@@ -411,6 +406,171 @@ def test_check_for_stalled_adopted_tasks_goes_in_ordered_fashion(self):
         assert executor.running == {key_2}
         assert executor.adopted_task_timeouts == {key_2: queued_dttm_2 + executor.task_adoption_timeout}
 
+    @pytest.mark.backend("mysql", "postgres")
+    @pytest.mark.parametrize(
+        "state, queued_dttm, executor_id",
+        [
+            (State.SCHEDULED, timezone.utcnow() - timedelta(days=2), '231'),
+            (State.QUEUED, timezone.utcnow(), '231'),
+            (State.QUEUED, timezone.utcnow(), None),
+        ],
+    )
+    def test_stuck_queued_tasks_are_cleared(
+        self, state, queued_dttm, executor_id, session, dag_maker, create_dummy_dag, create_task_instance
+    ):
+        """Test that clear_stuck_queued_tasks works"""
+        ti = create_task_instance(state=State.QUEUED)
+        ti.queued_dttm = queued_dttm
+        ti.external_executor_id = executor_id
+        session.merge(ti)
+        session.flush()
+        executor = celery_executor.CeleryExecutor()
+        executor._clear_stuck_queued_tasks()
+        session.flush()
+        ti = session.query(TaskInstance).filter(TaskInstance.task_id == ti.task_id).one()
+        assert ti.state == state
+
+    @pytest.mark.backend("mysql", "postgres")
+    def test_task_in_queued_tasks_dict_are_not_cleared(
+        self, session, dag_maker, create_dummy_dag, create_task_instance
+    ):
+        """Test that clear_stuck_queued_tasks doesn't clear tasks in executor.queued_tasks"""
+        ti = create_task_instance(state=State.QUEUED)
+        ti.queued_dttm = timezone.utcnow() - timedelta(days=2)
+        ti.external_executor_id = '231'
+        session.merge(ti)
+        session.flush()
+        executor = celery_executor.CeleryExecutor()
+        executor.queued_tasks = {ti.key: AsyncResult("231")}
+        executor._clear_stuck_queued_tasks()
+        session.flush()
+        ti = session.query(TaskInstance).filter(TaskInstance.task_id == ti.task_id).one()
+        assert ti.state == State.QUEUED
+
+    @pytest.mark.backend("mysql", "postgres")
+    def test_task_in_running_dict_are_not_cleared(
+        self, session, dag_maker, create_dummy_dag, create_task_instance
+    ):
+        """Test that clear_stuck_queued_tasks doesn't clear tasks in executor.running"""
+        ti = create_task_instance(state=State.QUEUED)
+        ti.queued_dttm = timezone.utcnow() - timedelta(days=2)
+        ti.external_executor_id = '231'
+        session.merge(ti)
+        session.flush()
+        executor = celery_executor.CeleryExecutor()
+        executor.running = {ti.key: AsyncResult("231")}
+        executor._clear_stuck_queued_tasks()
+        session.flush()
+        ti = session.query(TaskInstance).filter(TaskInstance.task_id == ti.task_id).one()

Review comment:
       ```
   assert executor.running == {ti.key: AsyncResult("231")}
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] jedcunningham commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1031696431


   Yep, exactly!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1042937424


   Thanks @kristoffern. Please can you update with the recent changes in #21556


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1029808201


   Increase the configuration to maybe every 1hr, or daily. I think that it runs too often and that's what's causing the spike:
   `AIRFLOW__CELERY__STUCK_QUEUED_TASK_CHECK_INTERVAL=86400`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r782877828



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +384,52 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session=None):

Review comment:
       ```suggestion
       def _clear_stuck_queued_tasks(self: Session = NEW_SESSION) -> None:
   ```
   
   while we are at it and since it is a new method




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy edited a comment on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy edited a comment on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1029808201


   Increase the configuration to maybe every 1hr, or daily. I think that it runs too often and that's what's causing the spike:
   `AIRFLOW__CELERY__STUCK_QUEUED_TASK_CHECK_INTERVAL=3600`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r782876872



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +384,52 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session=None):
+        """
+        Tasks can get stuck in queued state in DB while still not in
+        worker. This happens when the worker is autoscaled down and
+        the task is queued but has not been picked up by any worker prior to the scaling.
+
+        In such situation, we update the task instance state to scheduled so that
+        it can be queued again. We chose to use task_adoption_timeout to decide
+        """
+        if not isinstance(app.backend, DatabaseBackend):
+            # We only want to do this for database backends where
+            # this case has been spotted
+            return
+        # We use this instead of using bulk_state_fetcher because we
+        # may not have the stuck task in self.tasks and we don't want
+        # to clear task in self.tasks too
+        session_ = app.backend.ResultSession()
+        task_cls = getattr(app.backend, "task_cls", TaskDb)
+        with session_cleanup(session_):
+            celery_task_ids = [t.task_id for t in session_.query(task_cls.task_id).all()]
+        self.log.debug("Checking for stuck queued tasks")
+
+        max_allowed_time = utcnow() - self.task_adoption_timeout
+
+        for task in session.query(TaskInstance).filter(
+            TaskInstance.state == State.QUEUED, TaskInstance.queued_dttm < max_allowed_time
+        ):
+
+            self.log.info("Checking task %s", task)

Review comment:
       ```suggestion
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r755115318



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +390,36 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session=None):
+        """
+        For some reasons, tasks can get stuck in queued state in DB while still not in
+        self.queued_tasks and not in self.running_tasks.
+
+        In such situation, we update the task instance state to scheduled so that
+        it can be queued again. We chose to use task_adoption_timeout to decide
+        """
+        self.log.debug("Checking for stuck queued tasks")
+        queued_tasks = session.query(TaskInstance).filter(TaskInstance.state == State.QUEUED).all()
+
+        for task in queued_tasks:
+
+            self.log.info("Checking task %s", task)
+
+            if task.external_executor_id:
+                continue
+            if task.key in self.queued_tasks or task.key in self.running:
+                continue
+            # We use the task_adoption_timeout to decide if the task is stuck
+            if task.queued_dttm > utcnow() - datetime.timedelta(seconds=self.task_adoption_timeout):
+                continue

Review comment:
       You can do most of these at the SQL level and have the query only return the rows that should be cleared.
   
   But we probably need to _only_ do tasks owned by "this" executor.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] pingzh commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
pingzh commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1080169023


   @ephraimbuddy since we switched to SQS as our message broker from redis (it only happened when under high load) 2 years, we haven't experienced this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1046551295


   > Thanks @kristoffern. Please can you update with the recent changes in #21556
   
   Any update?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1041408115


   Thanks for the update, the error is fine and can be silenced because it's not much of importance. 
   I added the timeout to avoid maxing out the CPUs.
   Could you leave it to run for some days? I will update the PR to silence the error


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] Chris7 commented on a change in pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
Chris7 commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r824063695



##########
File path: airflow/executors/celery_executor.py
##########
@@ -337,6 +343,8 @@ def sync(self) -> None:
 
         if self.adopted_task_timeouts:
             self._check_for_stalled_adopted_tasks()
+        if time.time() - self.stuck_tasks_last_check_time > self.stuck_queued_task_check_interval:

Review comment:
       Won't this keep getting called every sync cycle? Shouldn't `stuck_tasks_last_check_time` be updated to the current time here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy merged pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy merged pull request #19769:
URL: https://github.com/apache/airflow/pull/19769


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] jedcunningham commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1031559032


   @kristoffern, can you try with the optimization from #21335 to see if that resolves the cpu usage issue? Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1041413651


   > 1: Patch 19769
   > 2: Patch 21335
   > 3: Patch 21556
   
   That's right. Monitor how often the webserver brings up scheduler does not appear to be running message.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1033663608


   @ephraimbuddy @jedcunningham I tested by adding the second patch as well into the deployment.
   
   It deployed without a problem, and performance-wise it seems fine at the moment (I've run a full pipeline of our ETL), but since the incident this weekend occurred after +24h I'm going to leave it running and see how it is tomorrow, so I really can't say at the moment any more than that.
   
   However there is a problem since I now have two tasks stuck in "Queued" for close to 3hours, just like the symptoms before the fix in this PR. Before applying the second patch I didn't see that problem with 19769 only, but I could of course just gotten "lucky" and the problem might just have become much more rare. Regardless it occurred now on the first attempt with path 19769 and 21335.
   
   ![image](https://user-images.githubusercontent.com/2227006/153191743-5ca8e82b-8a03-44f1-ad15-2995e069e4f3.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1041251743


   @ephraimbuddy I deployed to our staging cluster with the following patches applied in this order:
   
   1: Patch 19769
   2: Patch 21335
   3: Patch 21556
   
   Our pipeline is running, but I've seen at least once this warning in the webserver GUI.
   ![image](https://user-images.githubusercontent.com/2227006/154228349-a21225d7-86be-45f8-b701-75e13f635930.png)
   
   And after ~45min of running these errors started showing in the log output for the scheduler:
   ![image](https://user-images.githubusercontent.com/2227006/154228454-5017495e-11e9-4313-8e34-b9dcd6f73e93.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r756026593



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +387,43 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session=None):
+        """
+        For some reasons, tasks can get stuck in queued state in DB while still not in
+        self.queued_tasks and not in self.running_tasks.
+
+        In such situation, we update the task instance state to scheduled so that
+        it can be queued again. We chose to use task_adoption_timeout to decide
+        """
+        self.log.debug("Checking for stuck queued tasks")
+        queued_tasks = session.query(TaskInstance).filter(TaskInstance.state == State.QUEUED).all()
+
+        for task in queued_tasks:
+
+            self.log.info("Checking task %s", task)
+
+            if task.key in self.queued_tasks or task.key in self.running or not task.external_executor_id:
+                continue
+            # The pending state means task is waiting for execution or unknown
+            if AsyncResult(id=task.external_executor_id, app=app).state != 'PENDING':
+                continue
+            # We use the task_adoption_timeout to decide if the task is stuck
+            max_allowed_time = utcnow() - self.task_adoption_timeout
+
+            if max_allowed_time > task.queued_dttm:
+                # Using the if condition above instead of adding to query so that the logging below is correct
+                self.log.info(
+                    'TaskInstance: %s found in queued state for more than %s seconds, rescheduling',
+                    task,
+                    self.task_adoption_timeout.total_seconds(),
+                )
+                session.query(TaskInstance).filter(
+                    TaskInstance.dag_id == task.dag_id,
+                    TaskInstance.task_id == task.task_id,
+                    TaskInstance.run_id == task.run_id,
+                ).update({TaskInstance.state: State.SCHEDULED})

Review comment:
       Why not `task.state = State.SCHEDULED`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy edited a comment on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy edited a comment on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1033838250


   That means that the PR didn't fix the issue then cc: @jedcunningham 
   https://github.com/apache/airflow/issues/21225#issuecomment-1031543120


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1033679253


   And finally I should just mention that the Airflow webserver UI feels... sluggish and slower.
   That particular tab in browser has now twice had this error (no other tabs/websites in my browser, only the Airflow webserver tab).
   
   It's crashing with a SIGILL error and the page has to be reloaded.
   ![image](https://user-images.githubusercontent.com/2227006/153195534-a05e1fd6-a1e5-410c-ac9f-fa24ccc57012.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy edited a comment on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy edited a comment on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1041408115


   Thanks for the update, the error is fine and can be silenced because it's not much of importance. 
   I added the timeout to avoid maxing out the CPUs.
   Could you leave it to run for some days? I have updated the PR https://github.com/apache/airflow/pull/21556 to silence the error


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1029915119


   I do not think the operation is expensive, and I'm not sure if this caused the spike in CPU usage but let's observe it with the above configuration


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r756029280



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +387,43 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session=None):
+        """
+        For some reasons, tasks can get stuck in queued state in DB while still not in
+        self.queued_tasks and not in self.running_tasks.
+
+        In such situation, we update the task instance state to scheduled so that
+        it can be queued again. We chose to use task_adoption_timeout to decide
+        """
+        self.log.debug("Checking for stuck queued tasks")
+        queued_tasks = session.query(TaskInstance).filter(TaskInstance.state == State.QUEUED).all()
+
+        for task in queued_tasks:

Review comment:
       ```suggestion
           for task in session.query(TaskInstance).filter(TaskInstance.state == State.QUEUED):
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] tanelk commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r756306856



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +387,39 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session=None):
+        """
+        For some reasons, tasks can get stuck in queued state in DB while still not in
+        self.queued_tasks and not in self.running_tasks.

Review comment:
       Could you give here a little explanation, why tasks can get stuck. From the "For some reasons..." it sounds like there is some bug and this here is just a bandage.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r783667980



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +385,49 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session: Session = NEW_SESSION) -> None:
+        """
+        Tasks can get stuck in queued state in DB while still not in
+        worker. This happens when the worker is autoscaled down and
+        the task is queued but has not been picked up by any worker prior to the scaling.
+
+        In such situation, we update the task instance state to scheduled so that
+        it can be queued again. We chose to use task_adoption_timeout to decide
+        """
+        if not isinstance(app.backend, DatabaseBackend):
+            # We only want to do this for database backends where

Review comment:
       Oh yes, atleast in that case, the comment should be clear with a proper TODO or just solved in this PR too for all the other backends. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r782895730



##########
File path: tests/executors/test_celery_executor.py
##########
@@ -411,6 +406,171 @@ def test_check_for_stalled_adopted_tasks_goes_in_ordered_fashion(self):
         assert executor.running == {key_2}
         assert executor.adopted_task_timeouts == {key_2: queued_dttm_2 + executor.task_adoption_timeout}
 
+    @pytest.mark.backend("mysql", "postgres")
+    @pytest.mark.parametrize(
+        "state, queued_dttm, executor_id",
+        [
+            (State.SCHEDULED, timezone.utcnow() - timedelta(days=2), '231'),
+            (State.QUEUED, timezone.utcnow(), '231'),
+            (State.QUEUED, timezone.utcnow(), None),
+        ],
+    )
+    def test_stuck_queued_tasks_are_cleared(
+        self, state, queued_dttm, executor_id, session, dag_maker, create_dummy_dag, create_task_instance
+    ):
+        """Test that clear_stuck_queued_tasks works"""
+        ti = create_task_instance(state=State.QUEUED)
+        ti.queued_dttm = queued_dttm
+        ti.external_executor_id = executor_id
+        session.merge(ti)
+        session.flush()
+        executor = celery_executor.CeleryExecutor()
+        executor._clear_stuck_queued_tasks()
+        session.flush()
+        ti = session.query(TaskInstance).filter(TaskInstance.task_id == ti.task_id).one()
+        assert ti.state == state
+
+    @pytest.mark.backend("mysql", "postgres")
+    def test_task_in_queued_tasks_dict_are_not_cleared(
+        self, session, dag_maker, create_dummy_dag, create_task_instance
+    ):
+        """Test that clear_stuck_queued_tasks doesn't clear tasks in executor.queued_tasks"""
+        ti = create_task_instance(state=State.QUEUED)
+        ti.queued_dttm = timezone.utcnow() - timedelta(days=2)
+        ti.external_executor_id = '231'
+        session.merge(ti)
+        session.flush()
+        executor = celery_executor.CeleryExecutor()
+        executor.queued_tasks = {ti.key: AsyncResult("231")}
+        executor._clear_stuck_queued_tasks()
+        session.flush()
+        ti = session.query(TaskInstance).filter(TaskInstance.task_id == ti.task_id).one()

Review comment:
       based on the test name, you should check the following:
   
   ```
   assert executor.queued_tasks == {ti.key: AsyncResult("231")}
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r783662283



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +385,49 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session: Session = NEW_SESSION) -> None:
+        """
+        Tasks can get stuck in queued state in DB while still not in
+        worker. This happens when the worker is autoscaled down and
+        the task is queued but has not been picked up by any worker prior to the scaling.
+
+        In such situation, we update the task instance state to scheduled so that
+        it can be queued again. We chose to use task_adoption_timeout to decide
+        """
+        if not isinstance(app.backend, DatabaseBackend):
+            # We only want to do this for database backends where

Review comment:
       I can't figure how it could be solved in Redis. Do you have a reproductive step for Redis just like in #19699




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1025734470


   @ephraimbuddy I'm trying to test the patch locally, and I'm sure I'm doing something wrong, but after the cluster is deployed the scheduler crashes with the following error.
   
   ```
   file "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 54, in <module>
       from airflow.utils.session import NEW_SESSION, provide_session
   ImportError: cannot import name 'NEW_SESSION' from 'airflow.utils.session' (/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py)
   ```
   
   I've looked in this PR and can't find the NEW_SESSION in the files changed, and I also can't find it in Airflows main branch.
   I don't think I have access to the "astronomer:fix-stuck-task" repo/branch so I can't provide more details unfortunately.
   
   Am I missing something, or was something missedin the PR?
   
   I'm downloading this patch and applying it in my Dockerfile, I can provide the details if you want to.
   
   https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/19769.patch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1025813797


   @kristoffern The NEW_SESSION is a new addition that is not in 2.2.3, remove every reference to NEW_SESSION on the code and it should work.
   
   Also, remove the test changes before applying the patch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1064494801


   Since you are no longer having this issue, we should revert. I called your attention because of this issue: https://github.com/apache/airflow/issues/21225 which is related to this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] Chris7 commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
Chris7 commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1064404279


   Is there an issue you can link with respect to why it needs a reversion? I do recall having some issue w/ tasks being submitted to celery during a deployment and getting left in queued limbo, but unsure what about the current code is problematic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern edited a comment on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern edited a comment on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1041192298


   @ephraimbuddy I'll see if I can find some time today to test it. Just to double check, would that only be the #21556 patch then that should be applied, or should it be together with 
   
   1: Patch 19769
   2: Patch 21556
   
   Or would it include patch #21335 as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r755113791



##########
File path: airflow/executors/celery_executor.py
##########
@@ -337,6 +344,12 @@ def sync(self) -> None:
 
         if self.adopted_task_timeouts:
             self._check_for_stalled_adopted_tasks()
+        # Use EventScheduler to avoid checking this too often
+        self.event_scheduler = EventScheduler()
+        self.event_scheduler.call_regular_interval(
+            self.stuck_queued_task_check_interval,
+            self._clear_stuck_queued_tasks(),
+        )

Review comment:
       This isn't quite right -- it will re-create a new "timer" each time `sync()` is called, but never check it -- for that we have the instance live longer than one sync() call and to call `timers.run(blocking=False)`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r756334520



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +387,39 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session=None):
+        """
+        For some reasons, tasks can get stuck in queued state in DB while still not in
+        self.queued_tasks and not in self.running_tasks.

Review comment:
       Updated




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1029905879


   @ephraimbuddy Thanks I'll try it out and let you know. It currently defaults to every 5min right?
   Is the operation that expensive that it takes +5min when the system is under load?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1026656132


   @ephraimbuddy I've now deployed the modified patch-file to our Kubernetes cluster and tested since yesterday.
   
   In short I started all our parallel DAG runs for the data processing/ETL that we do.
   In the past it's reproduced the bug on virtually every attempt with tasks that get stuck in "Queued".
   
   At the moment I've ran it 4 times and on none of the runs have I seen the bug occur. 
   Now of course this isn't proof the bug is resolved, but it has drastically reduced the odds that the bug is still there :+1: 
   
   So to summarize:
   
   A: Patch doesn't break the system
   B: Patch by all accounts fixes the stuck in queued bug to occur.
   
   [19769-without-NEW_SESSION.patch.txt](https://github.com/apache/airflow/files/7977405/19769-without-NEW_SESSION.patch.txt)
   .
   
   Thank you for the hard work, and likely I will use this patch and modified Dockerfile in our deployment scripts until 2.2.4 or 2.3.0 is released :+1: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1025935520


   > Is NEW_SESSION part of 2.2.4 and 2.3.0 then?
   
   Yeah, will be.
   
   > I'll update here with a comment if you want to after I tested the patch šŸ‘
   
   I will appreciate that
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1038771962


   @kristoffern I have made an adjustment to this change, see https://github.com/apache/airflow/pull/21556, apply [the new changes](https://github.com/apache/airflow/pull/21556) and test?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1041172180


   > @kristoffern I have made an adjustment to this change, see #21556, apply [the new changes](https://github.com/apache/airflow/pull/21556) and test?
   
   Hi @kristoffern, any update on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1041192298


   @ephraimbuddy I'll see if I can find some time today to test it. Just to double check, would that only be the #21556 patch then that should be applied, or should it be together with 
   
   1: Patch 19769
   2: Patch 21556


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1046781023


   @ephraimbuddy Sorry, but I'm on limited amount of time currently for this and won't have any bandwidth to test it.
   Again, my apologies.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1041378128


   Final note. Our pipeline finished without any tasks stuck in queued in the end.
   
   Mid-way through I increased the CPU allocation for the scheduler from 1.5 => 3, but the AirflowTaskTimeout error kept showing up throughout the run. So it's not due to resource starvation on the scheduler end.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1042767445


   @ephraimbuddy Glad to report that the scheduler and system is still responsive +24h after the deployment.
   I've ran our full pipeline yesterday and now today without any hiccups.
   
   Looks like... all good... knock on wood...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on a change in pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r824145528



##########
File path: airflow/executors/celery_executor.py
##########
@@ -337,6 +343,8 @@ def sync(self) -> None:
 
         if self.adopted_task_timeouts:
             self._check_for_stalled_adopted_tasks()
+        if time.time() - self.stuck_tasks_last_check_time > self.stuck_queued_task_check_interval:

Review comment:
       Good call




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r756118870



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +387,43 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session=None):
+        """
+        For some reasons, tasks can get stuck in queued state in DB while still not in
+        self.queued_tasks and not in self.running_tasks.
+
+        In such situation, we update the task instance state to scheduled so that
+        it can be queued again. We chose to use task_adoption_timeout to decide
+        """
+        self.log.debug("Checking for stuck queued tasks")
+        queued_tasks = session.query(TaskInstance).filter(TaskInstance.state == State.QUEUED).all()
+
+        for task in queued_tasks:
+
+            self.log.info("Checking task %s", task)
+
+            if task.key in self.queued_tasks or task.key in self.running or not task.external_executor_id:
+                continue
+            # The pending state means task is waiting for execution or unknown
+            if AsyncResult(id=task.external_executor_id, app=app).state != 'PENDING':
+                continue
+            # We use the task_adoption_timeout to decide if the task is stuck
+            max_allowed_time = utcnow() - self.task_adoption_timeout
+
+            if max_allowed_time > task.queued_dttm:
+                # Using the if condition above instead of adding to query so that the logging below is correct
+                self.log.info(
+                    'TaskInstance: %s found in queued state for more than %s seconds, rescheduling',
+                    task,
+                    self.task_adoption_timeout.total_seconds(),
+                )
+                session.query(TaskInstance).filter(
+                    TaskInstance.dag_id == task.dag_id,
+                    TaskInstance.task_id == task.task_id,
+                    TaskInstance.run_id == task.run_id,
+                ).update({TaskInstance.state: State.SCHEDULED})

Review comment:
       I will try that, copied code :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r756022501



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +390,36 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session=None):
+        """
+        For some reasons, tasks can get stuck in queued state in DB while still not in
+        self.queued_tasks and not in self.running_tasks.
+
+        In such situation, we update the task instance state to scheduled so that
+        it can be queued again. We chose to use task_adoption_timeout to decide
+        """
+        self.log.debug("Checking for stuck queued tasks")
+        queued_tasks = session.query(TaskInstance).filter(TaskInstance.state == State.QUEUED).all()
+
+        for task in queued_tasks:
+
+            self.log.info("Checking task %s", task)
+
+            if task.external_executor_id:
+                continue
+            if task.key in self.queued_tasks or task.key in self.running:
+                continue
+            # We use the task_adoption_timeout to decide if the task is stuck
+            if task.queued_dttm > utcnow() - datetime.timedelta(seconds=self.task_adoption_timeout):
+                continue

Review comment:
       Since the external_executor_id of the task instance is no longer available in celery is there a way I could get only for the tasks in this executor? I'm currently searching broadly using AsyncResult to see if another executor is running this task




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kristoffern commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
kristoffern commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1029254332


   @ephraimbuddy A short update on the patch you probably want to know about.
   In the airflow-webserver I keep seeing these warnings as soon as I starts even a minor/trivial job/DAG.
   
   ![image](https://user-images.githubusercontent.com/2227006/152400799-024cb08a-8c3e-41ae-89a7-35bc2fbeec1b.png)
   
   Within the minute the warning disappears as the web-server gets a ping from the scheduler, and then a few seconds later the warning is back again (the number in the warning never goes over 60second btw).
   
   To clarify, all the DAGs and tasks are running just fine and are being scheduled. 
   If I had to guess the scheduler (evidently) is doing it's job, it's just a little slow letting the rest of the system know it's alive.
   
   We will keep the patched Airflow server for now, and if something changes I'll let you know. 
   But like I said, I figured you and @ashb would want to know.
   
   Also, I'm at your disposal if there's any information/logs I can provide you with, or test something in our staging environment.
   The patch file I'm using is attached in my previous message that has the NEW_SESSION removed if you need it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] jedcunningham commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1035265220


   Based on the issues @kristoffern had, I'm going to kick this to 2.3.0 so we have more time to properly fix this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy commented on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1068765605


   @pingzh, Since you also experience this, could you help in testing it? We have these patches 
   1: Patch https://github.com/apache/airflow/pull/19769
   2: Patch https://github.com/apache/airflow/pull/21335
   3: Patch https://github.com/apache/airflow/pull/21556
   and on the verge of reverting all if no one tests them


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] SamWheating commented on a change in pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
SamWheating commented on a change in pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#discussion_r783313436



##########
File path: airflow/executors/celery_executor.py
##########
@@ -377,6 +385,49 @@ def _check_for_stalled_adopted_tasks(self):
             for key in timedout_keys:
                 self.change_state(key, State.FAILED)
 
+    @provide_session
+    def _clear_stuck_queued_tasks(self, session: Session = NEW_SESSION) -> None:
+        """
+        Tasks can get stuck in queued state in DB while still not in
+        worker. This happens when the worker is autoscaled down and
+        the task is queued but has not been picked up by any worker prior to the scaling.
+
+        In such situation, we update the task instance state to scheduled so that
+        it can be queued again. We chose to use task_adoption_timeout to decide
+        """
+        if not isinstance(app.backend, DatabaseBackend):
+            # We only want to do this for database backends where

Review comment:
       Interesting - we have seen this same issue when using a Redis results backend - is there a good explanation for why this would only happen with DB backends?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ephraimbuddy edited a comment on pull request #19769: Handle stuck queued tasks in Celery for db backend

Posted by GitBox <gi...@apache.org>.
ephraimbuddy edited a comment on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1029808201


   Increase the configuration to maybe every 1hr, or daily. I think that it runs too often and that's what's causing the spike:
   `AIRFLOW__CELERY__STUCK_QUEUED_TASK_CHECK_INTERVAL=60`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #19769: Handle stuck queued tasks in Celery

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19769:
URL: https://github.com/apache/airflow/pull/19769#issuecomment-1006316891


   The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org