You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/20 19:41:10 UTC
[GitHub] [airflow] dstandish opened a new issue #14338: subdag tasks don't get picked up when dag concurrency set to 1
dstandish opened a new issue #14338:
URL: https://github.com/apache/airflow/issues/14338
They do get picked up like once or twice but pretty quickly somehow things get into a state where the scheduler thinks there is a task runninging in the dag (the subdag operator) so it won't schedule any more tasks (i.e. the real tasks, in the subdag)
i know [subdags are on the way out](https://github.com/apache/airflow/issues/12292) and i am no fan of subdags (i took this opportunity to remove all of them for our team) but it looks like a bug so i'm reporting it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] halilduygulu commented on issue #14338: subdag tasks don't get picked up when dag concurrency set to 1
Posted by GitBox <gi...@apache.org>.
halilduygulu commented on issue #14338:
URL: https://github.com/apache/airflow/issues/14338#issuecomment-783315440
I think in airflow 2 subdags are considered as a task instance and a dagrun, so when i have 15 subdags in a dag and set scheduler level general concurrency to 15, I have no running task instance slot left for actual tasks since subdag operator is holding up scheduler to create task instances.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #14338: subdag tasks don't get picked up when dag concurrency set to 1
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #14338:
URL: https://github.com/apache/airflow/issues/14338#issuecomment-783557419
2.0.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ernest-kr commented on issue #14338: subdag tasks don't get picked up when dag concurrency set to 1
Posted by GitBox <gi...@apache.org>.
ernest-kr commented on issue #14338:
URL: https://github.com/apache/airflow/issues/14338#issuecomment-783548234
which version of the airflow are you using ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #14338: subdag tasks don't get picked up when dag concurrency set to 1
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #14338:
URL: https://github.com/apache/airflow/issues/14338#issuecomment-783496230
the dags (where the subdags lived) had a dag concurrency of 1. it wasn't general scheduler concurrency (i.e. cluster parallelism or whatever it is)
but so it sounds like you are saying these simply needed to be updated to have a concurrency of 2.... for example. these seems a bit risky what if after finishing first task in a subdag the scheduler launches a new subdag instead of another task in the first subdag.... not that this happens just thinking it through.
anyway i don't see any change to concurrency handling in https://github.com/apache/airflow/blob/master/UPDATING.md .... and these subdag dags did run properly before upgrading, so it seems there may be something undesirable going on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on issue #14338: subdag tasks don't get picked up when dag concurrency set to 1
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #14338:
URL: https://github.com/apache/airflow/issues/14338#issuecomment-783496230
the dags (where the subdags lived) had a dag concurrency of 1. it wasn't general scheduler concurrency (i.e. cluster parallelism or whatever it is)
but so it sounds like you are saying these simply needed to be updated to have a concurrency of 2.... for example. these seems a bit risky what if after finishing first task in a subdag the scheduler launches a new subdag instead of another task in the first subdag.... not that this happens just thinking it through.
anyway i don't see any change to concurrency handling in https://github.com/apache/airflow/blob/master/UPDATING.md .... and these subdag dags did run properly before upgrading, (though there is mention of scheduler vs backfill) so it seems there may be something undesirable going on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org