Re: On some GUI tools for building Flink Streaming data flows...

[mnxfst <mn...@gmail.com>]

Hi Gyula, hi Slim,

being the core developer of SPQR, I am very glad to see the framework
referenced
on the Apache Flink dev list ;-)

Following the same approach as Flink - using real data streaming instead of
mini-batching - SPQR
adds some ad hoc'ness features to introduce more flexibility into the
streaming model. The
core idea is to shield away complex topology implementation and deployments
but provide 
a more business oriented layer for ad hoc queries and ad hoc stream
modificiation.

>From our point of view both features prepare stream processing for usage
within contexts that
are less tech oriented and where SQL is a broadly spoken language. But .. it
is not about 
refactoring SQL to stream processing (as I am not sure if a language
specified for static data is
suitable for streaming data) but to find a suitable approach which is
somehow similar to SQL
when it comes to adopting it in a business oriented context.

The most accurate metaphor for describing how we expect stream processing to
look like
for its users is that of a plumber working on a (more or less) large pipe
system. 

As both approaches are quite similar but Flink already provides some very
ice features that SPQR 
lacks, eg graph optimization & distribution and cluster support, I think it
may be worth checking if 
porting our vision of ad hoc'ness into Flink is possible - maybe as a layer
on top of the  
framework or as more core integration. Therefore - if such a feature is of
any interest 
for you - I would like to offer support on that task.

As outlined above and as Slim also mentioned it regarding StreamFlow UI
providing a less tech oriented
but more business oriented layer may lay ground for a broad adoption of
Flink since NONE of the
existing frameworks provides such a feature. All the other kids on the block
require heavy code
compilation and deployment before any work is done.

Cheers,
  Christian (@mnxfst)



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/On-some-GUI-tools-for-building-Flink-Streaming-data-flows-tp7233p7453.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: On some GUI tools for building Flink Streaming data flows...

[Stephan Ewen <se...@apache.org>]

Hi Christian!

Sounds like a very cool proposal, and it seems that Flink and SPQR could
complement each other very well.

If you are interested to see this on top of Flink, let's have a chat to see
how to best get started with this - what would be the easiest points of
integration.

As said, Flink cannot support dynamic topology changes at this point, but
interestingly, a lot of the building blocks for dynamic topology growths
are in place in the lower runtime levels, so even that could be added in
the future.

Greetings,
Stephan


On Tue, Aug 11, 2015 at 12:44 PM, Márton Balassi <ba...@gmail.com>
wrote:

> Hey Christian,
>
> Thanks for the insider view on SPQR. I have to agree with Gyula that
> dynamic topology build is not the highest priority for Flink currently, but
> certainly a very interesting feature and one that has already been
> requested by a couple of users.
>
> As for none of the open source streaming systems supporting a UI: with the
> open sourcing project Apex Datatorrent definitely has merit in this field,
> although I have to admit I do not exactly know how much of their platform
> is open sourced currently.
>
> Best,
>
> Marton
>
>
> On Tue, Aug 11, 2015 at 12:04 PM, mnxfst <mn...@gmail.com> wrote:
>
> > mnxfst
>

Re: On some GUI tools for building Flink Streaming data flows...

[Márton Balassi <ba...@gmail.com>]

Hey Christian,

Thanks for the insider view on SPQR. I have to agree with Gyula that
dynamic topology build is not the highest priority for Flink currently, but
certainly a very interesting feature and one that has already been
requested by a couple of users.

As for none of the open source streaming systems supporting a UI: with the
open sourcing project Apex Datatorrent definitely has merit in this field,
although I have to admit I do not exactly know how much of their platform
is open sourced currently.

Best,

Marton


On Tue, Aug 11, 2015 at 12:04 PM, mnxfst <mn...@gmail.com> wrote:

> mnxfst