You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/21 11:09:33 UTC
[GitHub] [airflow] dinigo opened a new issue #10443: tasks not being queued at the correct pool
dinigo opened a new issue #10443:
URL: https://github.com/apache/airflow/issues/10443
**Apache Airflow version**: docker image `apache/airflow:master-python3.8`
**What happened**: tasks not being scheduled to the correct pool
**What you expected to happen**: If the pool for the task is set to `subdag_pool`, for the task to be queued at this pool.
**How to reproduce it**:
Subdag tasks take all the running slots in the current pool, so I created a separate pool for the sub-dag tasks.

And then assigned this pool to the subdags
```python
brand_subdag_operator = SubDagOperator(
task_id=brand_id,
subdag=brand_subdag,
pool='subdags_pool',
)
```
But these tasks keep being scheduled at the `default_pool` instead of the new `subdag_pool`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dinigo edited a comment on issue #10443: tasks not being queued at the correct pool
Posted by GitBox <gi...@apache.org>.
dinigo edited a comment on issue #10443:
URL: https://github.com/apache/airflow/issues/10443#issuecomment-760264354
@potiuk I cannot check right now if this is still happening, sorry. I was using 2.0 from the master branch. I want to mention that I solved this issue setting the SubDagOperator (which is a sensor under the hood) to `mode = 'reschedule'` so it doesn't take a whole slot.
Is there a reason for the `'reschedule'` mode not being the default? I have enough sensors to fill and block the pool. That's the source of this issue actually.
Anyway I'm using less sensors and building bigger dags now that we have TaskGroups.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #10443: tasks not being queued at the correct pool
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #10443:
URL: https://github.com/apache/airflow/issues/10443
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10443: tasks not being queued at the correct pool
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10443:
URL: https://github.com/apache/airflow/issues/10443#issuecomment-759404480
In 1.10 all subdags are scheduled in the same workers/pools as the parent DAGs - by design. This has been changed in Airflow 2.0 where SubDags are processed in a different way and they can run in different workers/pools.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dinigo commented on issue #10443: tasks not being queued at the correct pool
Posted by GitBox <gi...@apache.org>.
dinigo commented on issue #10443:
URL: https://github.com/apache/airflow/issues/10443#issuecomment-760264354
@potiuk I cannot check right now if this is still happening, sorry. I want to mention that I solved this issue setting the SubDagOperator (which is a sensor under the hood) to `mode = 'reschedule'` so it doesn't take a whole slot.
Is there a reason for the `'reschedule'` mode not being the default? I have enough sensors to fill and block the pool. That's the source of this issue actually.
Anyway I'm using less sensors and building bigger dags now that we have TaskGroups.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] maierru commented on issue #10443: tasks not being queued at the correct pool
Posted by GitBox <gi...@apache.org>.
maierru commented on issue #10443:
URL: https://github.com/apache/airflow/issues/10443#issuecomment-759319931
version `1.10.12` same behaviour
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org