You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by D G <da...@ontrenet.com> on 2022/09/15 15:41:57 UTC

LambdaFlow: Python Nifi Clone

Hi,
  Here is a look at LambdaFlow, a Nifi inspired, lambda dataflow platform in python. I share this to Nifi community as LambdaFlow is a re-imagining of Nifi as a fully distributed, at-scale, data flow system that Nifi might be able to derive improvements from.

Key difference is processors in LambdaFlow run at-scale, as clustered python functions bound to multiple CPUs across servers and can stretch on-demand. This turns a processor into its own clustered service transparent to the lambda functions. Also, there are no flow files, you work directly with python code and results (usually JSON) are forwarded along the dataflow topology using transactional messaging lower in the stack.

Enjoy!

https://www.youtube.com/watch?v=vjQCQwIVodg

[cid:86263568-9812-42c2-9d79-06a671f25cd6]

Re: LambdaFlow: Python Nifi Clone

Posted by Mike Thomsen <mi...@gmail.com>.
Interesting project, but please be mindful of the fact that this is not
part of the Apache NiFi suite of products and these mailing lists are
intended to support Apache NiFi when posting further updates.

Thanks,

Mike

On Thu, Sep 15, 2022 at 3:38 PM D G <da...@ontrenet.com> wrote:

> It's in a private github repo still until I get the community codebase
> finished. Then I'll release it public. The UI is written in Node/VueJS and
> is fully extensible and customizable separate from backend.
>
> Sent from my Verizon, Samsung Galaxy smartphone
> Get Outlook for Android <https://aka.ms/AAb9ysg>
> ------------------------------
> *From:* Jean-Sebastien Vachon <js...@brizodata.com>
> *Sent:* Thursday, September 15, 2022 3:26:53 PM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: LambdaFlow: Python Nifi Clone
>
> Wow. this is very interesting. Is this project OpenSource? I couldn't find
> it on GitHub.
>
>
>
> Jean-Sébastien Vachon
> ARCHITECT & CO-FOUNDER
>  jsvachon@brizodata.com
>  brizodata.com • [image: LinkedIn]
> <https://www.linkedin.com/company/brizodata>
> <https://brizodata.com>
> ------------------------------
> *From:* D G <da...@ontrenet.com>
> *Sent:* Thursday, September 15, 2022 11:41 AM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* LambdaFlow: Python Nifi Clone
>
> Hi,
>   Here is a look at LambdaFlow, a Nifi inspired, lambda dataflow platform
> in python. I share this to Nifi community as LambdaFlow is a re-imagining
> of Nifi as a fully distributed, at-scale, data flow system that Nifi might
> be able to derive improvements from.
>
> Key difference is processors in LambdaFlow run at-scale, as clustered
> python functions bound to multiple CPUs across servers and can stretch
> on-demand. This turns a processor into its own clustered service
> transparent to the lambda functions. Also, there are no flow files, you
> work directly with python code and results (usually JSON) are forwarded
> along the dataflow topology using transactional messaging lower in the
> stack.
>
> Enjoy!
>
> https://www.youtube.com/watch?v=vjQCQwIVodg
>
>
>

Re: LambdaFlow: Python Nifi Clone

Posted by D G <da...@ontrenet.com>.
It's in a private github repo still until I get the community codebase finished. Then I'll release it public. The UI is written in Node/VueJS and is fully extensible and customizable separate from backend.

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Jean-Sebastien Vachon <js...@brizodata.com>
Sent: Thursday, September 15, 2022 3:26:53 PM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: LambdaFlow: Python Nifi Clone

Wow. this is very interesting. Is this project OpenSource? I couldn't find it on GitHub.



Jean-Sébastien Vachon
ARCHITECT & CO-FOUNDER
[https://brizofoodmet.icns.email/icon/fa-light-16-envelope-666666.png] jsvachon@brizodata.com<ma...@brizodata.com>
[https://brizofoodmet.icns.email/icon/fa-light-16-arrowright-666666.png] brizodata.com<https://brizodata.com> • [LinkedIn] <https://www.linkedin.com/company/brizodata>


[https://brizofoodmet.img.email/images/62a0e8f024ec8.png]<https://brizodata.com>
________________________________
From: D G <da...@ontrenet.com>
Sent: Thursday, September 15, 2022 11:41 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: LambdaFlow: Python Nifi Clone

Hi,
  Here is a look at LambdaFlow, a Nifi inspired, lambda dataflow platform in python. I share this to Nifi community as LambdaFlow is a re-imagining of Nifi as a fully distributed, at-scale, data flow system that Nifi might be able to derive improvements from.

Key difference is processors in LambdaFlow run at-scale, as clustered python functions bound to multiple CPUs across servers and can stretch on-demand. This turns a processor into its own clustered service transparent to the lambda functions. Also, there are no flow files, you work directly with python code and results (usually JSON) are forwarded along the dataflow topology using transactional messaging lower in the stack.

Enjoy!

https://www.youtube.com/watch?v=vjQCQwIVodg

[cid:86263568-9812-42c2-9d79-06a671f25cd6]

Re: LambdaFlow: Python Nifi Clone

Posted by Jean-Sebastien Vachon <js...@brizodata.com>.
Wow. this is very interesting. Is this project OpenSource? I couldn't find it on GitHub.



Jean-Sébastien Vachon
ARCHITECT & CO-FOUNDER
[https://brizofoodmet.icns.email/icon/fa-light-16-envelope-666666.png] jsvachon@brizodata.com<ma...@brizodata.com>
[https://brizofoodmet.icns.email/icon/fa-light-16-arrowright-666666.png] brizodata.com<https://brizodata.com> • [LinkedIn] <https://www.linkedin.com/company/brizodata>


[https://brizofoodmet.img.email/images/62a0e8f024ec8.png]<https://brizodata.com>
________________________________
From: D G <da...@ontrenet.com>
Sent: Thursday, September 15, 2022 11:41 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: LambdaFlow: Python Nifi Clone

Hi,
  Here is a look at LambdaFlow, a Nifi inspired, lambda dataflow platform in python. I share this to Nifi community as LambdaFlow is a re-imagining of Nifi as a fully distributed, at-scale, data flow system that Nifi might be able to derive improvements from.

Key difference is processors in LambdaFlow run at-scale, as clustered python functions bound to multiple CPUs across servers and can stretch on-demand. This turns a processor into its own clustered service transparent to the lambda functions. Also, there are no flow files, you work directly with python code and results (usually JSON) are forwarded along the dataflow topology using transactional messaging lower in the stack.

Enjoy!

https://www.youtube.com/watch?v=vjQCQwIVodg

[cid:86263568-9812-42c2-9d79-06a671f25cd6]