You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/09 14:49:32 UTC

[GitHub] [airflow] brunorigal opened a new issue, #26271: multiple expand with parallelism maintained

brunorigal opened a new issue, #26271:
URL: https://github.com/apache/airflow/issues/26271

   ### Description
   
   I would like to build a graph like this:
   
                 ┌───►  task_2_1  ─────►  task_2_1
                 │
                 │
                 │
   task 1 ─────►  task_2_2  ─────►  task_3_2
                 │
                 │
                 │
                 │
                 └──►   task_2_N  ─────►  task_3_N
   
   
   Where the number of vertical tasks (N) is controlled dynamically by the first step computation. A possible solution would be:
   
   ```python
   from airflow import DAG
   from airflow.decorators import task, task_group
   from pendulum import datetime, now
   
   
   @task
   def task_1():
       return list(range(5))
   
   
   @task
   def task_2(task_num):
       return task_num
   
   
   @task
   def task_3(task_num):
       return task_num
   
   
   with DAG(dag_id="my_dag", start_date=now(), schedule_interval=None) as dag:
       task_3.expand(task_num=task_2.expand(task_num=task_1()))
   ```
   
   But the task_3 instances start to run only when all the task_2 instance have finished. I would rather group tasks by instances of task_2 and task_3 sharing the same task_num.
   An intuitive solution would have been to use a taskgroup, but these do not have the expand method.
   
   ### Use case/motivation
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr closed issue #26271: multiple expand with parallelism maintained

Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #26271: multiple expand with parallelism maintained
URL: https://github.com/apache/airflow/issues/26271


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #26271: multiple expand with parallelism maintained

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #26271:
URL: https://github.com/apache/airflow/issues/26271#issuecomment-1243768114

   Duplicate of https://github.com/apache/airflow/issues/25032


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #26271: multiple expand with parallelism maintained

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #26271:
URL: https://github.com/apache/airflow/issues/26271#issuecomment-1243457854

   Is this essentially depth-first execution a la #25032?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] brunorigal commented on issue #26271: multiple expand with parallelism maintained

Posted by GitBox <gi...@apache.org>.
brunorigal commented on issue #26271:
URL: https://github.com/apache/airflow/issues/26271#issuecomment-1243626144

   Yes indeed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #26271: multiple expand with parallelism maintained

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26271:
URL: https://github.com/apache/airflow/issues/26271#issuecomment-1242072673

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org