You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by George Leslie-Waksman <ge...@cloverhealth.com> on 2016/10/01 05:36:05 UTC

Re: Split/Chunk operator

Airflow isn't really designed for tasks to pass data to each other. There
are limited capabilities for doing so but Airflow is primarily a workflow
management tool, not a dataflow tool.

--George Leslie-Waksman (Clover Health)

On Sun, Sep 4, 2016 at 6:01 AM Mohsen Bande <mo...@gmail.com> wrote:

> it would be nice to have a split (python) operator, which takes a job that
> returns a list and streams each item to the specified job. like this
> pattern:http://www.workflowpatterns.com/patterns/control/basic/wcp2.php
> also, a chunk operator that splits a list to smaller chunks is more general
> solution
> is there any try on this already?
> any help would be highly appreciated
>