You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/01 17:42:26 UTC

[GitHub] [airflow] freephys opened a new issue #11224: Branch airflow in middle and test task with different parameters in parallel 

freephys opened a new issue #11224:
URL: https://github.com/apache/airflow/issues/11224


   Hi,
    I am looking for a way to achieve this inside airflow, hopefully some easy demo code to study with.  Start with several jobs with say 50 parameters, then at some point (After finishing task A->B) , I wanna to run task C in parallel with different values of say 5 parameters(those 5 parameters will try different values) , after running task C in parallel, all results are passed onto task D and E with same parameters.  The dag id etc are needed to be generated dynamically.
     
   Any idea about if this can be done with current airflow or a feature to be ask?
    
   The advantage will be saving time on task A and B and try task C with several different parameters so we can pick best one based on the results of task D
   
   thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #11224: Branch airflow in middle and test task with different parameters in parallel

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #11224:
URL: https://github.com/apache/airflow/issues/11224


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #11224: Branch airflow in middle and test task with different parameters in parallel

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #11224:
URL: https://github.com/apache/airflow/issues/11224#issuecomment-812574812


   I also don't understand the issue.
   In any case this sounds more like ETL design question so if you need support please use Stackoverflow or troubleshooting channel in Airflow slack.
   If this is a feature request please comment with more details and use case examples.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #11224: Branch airflow in middle and test task with different parameters in parallel

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11224:
URL: https://github.com/apache/airflow/issues/11224#issuecomment-702447468


   I do not fully understand the question. [XCOM](https://airflow.readthedocs.io/en/latest/concepts.html#xcoms) is used to pass the results of one task to the next.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #11224: Branch airflow in middle and test task with different parameters in parallel

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #11224:
URL: https://github.com/apache/airflow/issues/11224#issuecomment-702292884


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org