You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pedro-cf (via GitHub)" <gi...@apache.org> on 2024/10/10 11:09:28 UTC

[I] Task assigned with queue "heavy" landed on worker assigned to "default" queue [airflow]

pedro-cf opened a new issue, #42894:
URL: https://github.com/apache/airflow/issues/42894

   ### Apache Airflow version
   
   2.10.2
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   I'm currently using an airflow instance setup with `CeleryExecutor`.
   
   I have a task assigned to a "heavy" queue:
   
   ```python
       @task.virtualenv(
           trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS,
           requirements=[
               "my_package=1.1.1"
           ],
           index_urls=["my_index_url"],
           venv_cache_path="/tmp",
           queue="heavy"
       )
       def my_task(arg1, arg2, params=None):
           ...
   ```
   
   but for some reason the task is running on a worker configured to only handle the default queue?
   
   Worker Queues in flower:
   ![image](https://github.com/user-attachments/assets/64b9b6a8-2c9c-40f1-a55b-518452999852)
   
   DAG RUN details:
   ![image](https://github.com/user-attachments/assets/036ccc27-33eb-4047-861c-ca065eae8750)
   
   
   
   ### What you think should happen instead?
   
   Tasks assigned to a specific queue should only run in workers assigned to that specific queue.
   
   ### How to reproduce
   
   Not sure how.
   
   ### Operating System
   
   Ubuntu 22
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   `CeleryExecutor`
   2 workers:
   ```
   celery worker --concurrency ${DEFAULT_WORKER_CONCURRENCY}
   celery worker -q heavy --concurrency ${HEAVY_WORKER_CONCURRENCY}
   ```
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task assigned with queue "heavy" landed on worker assigned to "default" queue [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #42894:
URL: https://github.com/apache/airflow/issues/42894#issuecomment-2409029853

   > -> IMPORTANT: The DAG Run existed prior to the addition of the queue="heavy" to the dag's task. And the Task state was cleared in order to restart the task.
   
   
   That's the reason. Past dag-run queues cannot be changed after dag_run and specifically "task instance" object entry gets created.
   
   See https://airflow.apache.org/docs/apache-airflow/stable/database-erd-ref.html - the task_instance model contains "queue" so once task instance has the queue set there, re-running the same task instance will use what is there. 
   
   You could likely modify it manually, but currently there is no way - I think to modify it from UI - other than running backfill (@dstandish ?) which I think should do what you want.
   
   Which could be possibly a good idea to change in Airlfow 3. There are similar discussions happening about other features of backfill and pool behaviours at the devlist that might get improved in Airflow 3 -  https://lists.apache.org/thread/zbm6tvlcz62nc9hl1mzrzz9t4bcrjngc , https://lists.apache.org/thread/jmj842wsw78clk9twdrz1t71ogsbk10s and others - so if you think it's a good idea to introduce such feature, feel free to start a new thread at the devlist.
   
   Converting it into a discussion in case more discussion here is needed, but I encourage you to continue at the devlist.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Task assigned with queue "heavy" landed on worker assigned to "default" queue [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #42894: Task assigned with queue "heavy" landed on worker assigned to "default" queue
URL: https://github.com/apache/airflow/issues/42894


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org