Pipeline manager/scheduler frameworks

[Sridevi Nookala <sn...@parallelwireless.com>]

Hi,


Our analytics app has many data pipelines , some in python /java (using beam) etc,

Any suggestions for a pipeline manager/scheduler framework that manages/orchestrates these different pipelines.


thanks

Sri

Re: Pipeline manager/scheduler frameworks

[Adeel Ahmad <aa...@gmail.com>]

Airflow would be good but you will probably have to modify it to support
stream processing. Any DAG based manager would be useful in your case.
Luigi works too, but airflow has a sleeker UI.
You could also try streamsets. GCP provides composer which uses airflow and
dataflow for beam. AWS has Glue although might not fully cover your case.


On Sat, 9 Feb 2019 at 00:45, Rui Wang <ru...@google.com> wrote:

> Apache Airflow is a scheduling system that can help manage data pipelines.
> I have seen Airflow is used to manage a few thousand hive/spark/presto
> pipelines.
>
>
> -Rui
>
> On Fri, Feb 8, 2019 at 4:08 PM Sridevi Nookala <
> snookala@parallelwireless.com> wrote:
>
>> Hi,
>>
>>
>> Our analytics app has many data pipelines , some in python /java (using
>> beam) etc,
>>
>> Any suggestions for a pipeline manager/scheduler framework that
>> manages/orchestrates these different pipelines.
>>
>>
>> thanks
>>
>> Sri
>>
>

-- 
Thanks,

Adeel Ahmad
m: (+44) 7721724715
e: aahmad1811@gmail.com

Re: Pipeline manager/scheduler frameworks

[Rui Wang <ru...@google.com>]

Apache Airflow is a scheduling system that can help manage data pipelines.
I have seen Airflow is used to manage a few thousand hive/spark/presto
pipelines.


-Rui

On Fri, Feb 8, 2019 at 4:08 PM Sridevi Nookala <
snookala@parallelwireless.com> wrote:

> Hi,
>
>
> Our analytics app has many data pipelines , some in python /java (using
> beam) etc,
>
> Any suggestions for a pipeline manager/scheduler framework that
> manages/orchestrates these different pipelines.
>
>
> thanks
>
> Sri
>