On some GUI tools for building Flink Streaming data flows...

[Slim Baltagi <sb...@gmail.com>]

Hi

As Flink streaming API is being stabilized, it is worth investigating
tooling around it to make developer's life easier and also turning Flink
streaming a more attractive stream processing platform. 
I would like to suggest a couple tools which are worth integrating with
Flink streaming: 

1. 'StreamFlow™ is a stream processing tool designed to help build and
monitor processing workflows. The ultimate goal of StreamFlow is to make
working with stream processing frameworks such as Apache Storm easier,
faster, and with "enterprise" like management functionality.

StreamFlow also provides a mechanism for non-developers such as data
scientists, analysts, or operational users to rapidly build scalable data
flows and analytics.'

Reference:https://github.com/lmco/streamflow The topology builder looks
really slick!

Any chance of seeing StreamFlow a GUI for building Flink streaming data
flows? 

2. SPQR pronounced Spooker is a framework for building *dynamic * data
stream processing pipelines: a feature that is neither available in Apache
Storm nor in Spark Streaming. 
https://github.com/ottogroup/SPQR

It might be worth investigating SPQR, integrating it with Flink and make it
production ready. 

What are your thoughts? 

Thanks

Slim Baltagi



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/On-some-GUI-tools-for-building-Flink-Streaming-data-flows-tp7233.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: On some GUI tools for building Flink Streaming data flows...

[Stephan Ewen <se...@apache.org>]

Hi Christian!

Sounds like a very cool proposal, and it seems that Flink and SPQR could
complement each other very well.

If you are interested to see this on top of Flink, let's have a chat to see
how to best get started with this - what would be the easiest points of
integration.

As said, Flink cannot support dynamic topology changes at this point, but
interestingly, a lot of the building blocks for dynamic topology growths
are in place in the lower runtime levels, so even that could be added in
the future.

Greetings,
Stephan


On Tue, Aug 11, 2015 at 12:44 PM, Márton Balassi <ba...@gmail.com>
wrote:

> Hey Christian,
>
> Thanks for the insider view on SPQR. I have to agree with Gyula that
> dynamic topology build is not the highest priority for Flink currently, but
> certainly a very interesting feature and one that has already been
> requested by a couple of users.
>
> As for none of the open source streaming systems supporting a UI: with the
> open sourcing project Apex Datatorrent definitely has merit in this field,
> although I have to admit I do not exactly know how much of their platform
> is open sourced currently.
>
> Best,
>
> Marton
>
>
> On Tue, Aug 11, 2015 at 12:04 PM, mnxfst <mn...@gmail.com> wrote:
>
> > mnxfst
>

Re: On some GUI tools for building Flink Streaming data flows...

[Márton Balassi <ba...@gmail.com>]

Hey Christian,

Thanks for the insider view on SPQR. I have to agree with Gyula that
dynamic topology build is not the highest priority for Flink currently, but
certainly a very interesting feature and one that has already been
requested by a couple of users.

As for none of the open source streaming systems supporting a UI: with the
open sourcing project Apex Datatorrent definitely has merit in this field,
although I have to admit I do not exactly know how much of their platform
is open sourced currently.

Best,

Marton


On Tue, Aug 11, 2015 at 12:04 PM, mnxfst <mn...@gmail.com> wrote:

> mnxfst

Re: On some GUI tools for building Flink Streaming data flows...

[mnxfst <mn...@gmail.com>]

Hi Gyula, hi Slim,

being the core developer of SPQR, I am very glad to see the framework
referenced
on the Apache Flink dev list ;-)

Following the same approach as Flink - using real data streaming instead of
mini-batching - SPQR
adds some ad hoc'ness features to introduce more flexibility into the
streaming model. The
core idea is to shield away complex topology implementation and deployments
but provide 
a more business oriented layer for ad hoc queries and ad hoc stream
modificiation.

>From our point of view both features prepare stream processing for usage
within contexts that
are less tech oriented and where SQL is a broadly spoken language. But .. it
is not about 
refactoring SQL to stream processing (as I am not sure if a language
specified for static data is
suitable for streaming data) but to find a suitable approach which is
somehow similar to SQL
when it comes to adopting it in a business oriented context.

The most accurate metaphor for describing how we expect stream processing to
look like
for its users is that of a plumber working on a (more or less) large pipe
system. 

As both approaches are quite similar but Flink already provides some very
ice features that SPQR 
lacks, eg graph optimization & distribution and cluster support, I think it
may be worth checking if 
porting our vision of ad hoc'ness into Flink is possible - maybe as a layer
on top of the  
framework or as more core integration. Therefore - if such a feature is of
any interest 
for you - I would like to offer support on that task.

As outlined above and as Slim also mentioned it regarding StreamFlow UI
providing a less tech oriented
but more business oriented layer may lay ground for a broad adoption of
Flink since NONE of the
existing frameworks provides such a feature. All the other kids on the block
require heavy code
compilation and deployment before any work is done.

Cheers,
  Christian (@mnxfst)



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/On-some-GUI-tools-for-building-Flink-Streaming-data-flows-tp7233p7453.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: On some GUI tools for building Flink Streaming data flows...

[Gyula Fóra <gy...@apache.org>]

Hi Slim,

I totally agree with you that we should start working on supporting tools
like these.

StreamFlow is really nice tool, I also like it a lot. I think it would be
not too hard to integrate it with Flink as there are several projects that
have already used Flink streaming in a compositional way. (SAMOA, or the
Storm compatibility layer). So I guess this is just a matter of someone
picking it up at one point, but I am really looking forward to it , and
will support anyone doing it :)

As for SPQR and the dynamic pipelines, I don't think we have this on the
short term roadmap, but it would indeed be a nice feature. I don't see any
system limitations that would keep us from implementing this either once we
start working on dynamic modifications to the execution graph (such as for
automatic scaling)

Cheers,
Gyula

Slim Baltagi <sb...@gmail.com> ezt írta (időpont: 2015. júl. 30., Cs,
5:50):

> Hi
>
> As Flink streaming API is being stabilized, it is worth investigating
> tooling around it to make developer's life easier and also turning Flink
> streaming a more attractive stream processing platform.
> I would like to suggest a couple tools which are worth integrating with
> Flink streaming:
>
> 1. 'StreamFlow™ is a stream processing tool designed to help build and
> monitor processing workflows. The ultimate goal of StreamFlow is to make
> working with stream processing frameworks such as Apache Storm easier,
> faster, and with "enterprise" like management functionality.
>
> StreamFlow also provides a mechanism for non-developers such as data
> scientists, analysts, or operational users to rapidly build scalable data
> flows and analytics.'
>
> Reference:https://github.com/lmco/streamflow The topology builder looks
> really slick!
>
> Any chance of seeing StreamFlow a GUI for building Flink streaming data
> flows?
>
> 2. SPQR pronounced Spooker is a framework for building *dynamic * data
> stream processing pipelines: a feature that is neither available in Apache
> Storm nor in Spark Streaming.
> https://github.com/ottogroup/SPQR
>
> It might be worth investigating SPQR, integrating it with Flink and make it
> production ready.
>
> What are your thoughts?
>
> Thanks
>
> Slim Baltagi
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/On-some-GUI-tools-for-building-Flink-Streaming-data-flows-tp7233.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>