You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Mohsen Bande <mo...@gmail.com> on 2016/09/04 13:01:21 UTC

Split/Chunk operator

it would be nice to have a split (python) operator, which takes a job that
returns a list and streams each item to the specified job. like this
pattern:http://www.workflowpatterns.com/patterns/control/basic/wcp2.php
also, a chunk operator that splits a list to smaller chunks is more general
solution
is there any try on this already?
any help would be highly appreciated

Re: Split/Chunk operator

Posted by George Leslie-Waksman <ge...@cloverhealth.com>.
Airflow isn't really designed for tasks to pass data to each other. There
are limited capabilities for doing so but Airflow is primarily a workflow
management tool, not a dataflow tool.

--George Leslie-Waksman (Clover Health)

On Sun, Sep 4, 2016 at 6:01 AM Mohsen Bande <mo...@gmail.com> wrote:

> it would be nice to have a split (python) operator, which takes a job that
> returns a list and streams each item to the specified job. like this
> pattern:http://www.workflowpatterns.com/patterns/control/basic/wcp2.php
> also, a chunk operator that splits a list to smaller chunks is more general
> solution
> is there any try on this already?
> any help would be highly appreciated
>