You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/26 23:51:40 UTC

[GitHub] [airflow] casassg opened a new issue #11870: Add @taskgroup decorator

casassg opened a new issue #11870:
URL: https://github.com/apache/airflow/issues/11870


   **Description**
   
   Similarly to `@dag` and `@task` decorators, we can add a decorator to easily generate `TaskGroup` instances. 
   
   **Use case / motivation**
   
   This would be used for reusable pieces of DAGs. An example may be to hyperparameter tune an ML pipeline. You may want to generate several parallel subpipelines with different settings/parameters to be executed in parallel. At the end pull the result and decide the best parametrization.  
   
   ```py
   @task
   collect_dataset(...)
     pass
   
   @task
   def train_model(...)
      pass
   
   @task_group
   def movielens_model_pipeline(learning_rate: int, feature_list: list, dataset: XComArg):  
     dataset = filter_dataset(dataset, feature_list)
     train_model(dataset, learning_rate)
   
   @dag
   def movielens_hpt(dataset_path: str, feature_list:list=['year', 'director']):
     dataset = load_dataset(dataset_path)
     for i in range(0.1, 0.9, step=0.1):
       movielens_model_pipeline(i, feature_list, dataset)
     
     decide_best_model(...)
   ```
   
   
   
   
   **Related Issues**
   
   https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3AAIP-31
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] thcidale0808 commented on issue #11870: Add @taskgroup decorator

Posted by GitBox <gi...@apache.org>.
thcidale0808 commented on issue #11870:
URL: https://github.com/apache/airflow/issues/11870#issuecomment-721988950


   @ryw can I implement this feature?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ryw commented on issue #11870: Add @taskgroup decorator

Posted by GitBox <gi...@apache.org>.
ryw commented on issue #11870:
URL: https://github.com/apache/airflow/issues/11870#issuecomment-717317435


   Sounds good to me 👍 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ryw commented on issue #11870: Add @taskgroup decorator

Posted by GitBox <gi...@apache.org>.
ryw commented on issue #11870:
URL: https://github.com/apache/airflow/issues/11870#issuecomment-724256479


   @thcidale0808 sorry for slow reply - looks like someone else is working on it here - https://github.com/apache/airflow/pull/12159


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] thcidale0808 commented on issue #11870: Add @taskgroup decorator

Posted by GitBox <gi...@apache.org>.
thcidale0808 commented on issue #11870:
URL: https://github.com/apache/airflow/issues/11870#issuecomment-724289880


   thanks @ryw. I'll look for another one to contribute. @kaxil maybe you should assign this issue to VBhojawala.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dimberman closed issue #11870: Add @taskgroup decorator

Posted by GitBox <gi...@apache.org>.
dimberman closed issue #11870:
URL: https://github.com/apache/airflow/issues/11870


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] VBhojawala commented on issue #11870: Add @taskgroup decorator

Posted by GitBox <gi...@apache.org>.
VBhojawala commented on issue #11870:
URL: https://github.com/apache/airflow/issues/11870#issuecomment-724436483


   Closing current pull request to re-raise pull request. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] VBhojawala edited a comment on issue #11870: Add @taskgroup decorator

Posted by GitBox <gi...@apache.org>.
VBhojawala edited a comment on issue #11870:
URL: https://github.com/apache/airflow/issues/11870#issuecomment-724436483


   Closing current pull request to re-raise pull request from branch instead of fork itself. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org