You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Palle <pa...@sport.dk> on 2016/05/29 18:20:04 UTC

Visualize result of Flink job

Hi there

I am using Flink to analyse a lot of incoming data. Every 10 seconds it makes sense to present the analysis so far as some form of visualization. Every 10 seconds I therefore will replace the current contents of the visualization/presentation with the analysis result of the most recent 10 seconds.

I was first thinking of using ElasticSearch/Kibana for this because I know it should be easy to set up, but I am thinking it may not be the best fit, because Elastic is by nature a search engine that is good for trending and stuff like that - not entire replacement of the current view. And therefore I may also experience difficulties implementing the view in Elastic.

Does anyone know of any other visualization tools that work well with Flink? ...where it is easy to export the result of a Flink job to a user interface (web).

Thanks
Palle

Re: Visualize result of Flink job

Posted by Robert Metzger <rm...@apache.org>.
Hi,
sounds doable. I think it should be easy to set up a first proof of concept
for this.
Let us know if you need any further assistance.

Regards,
Robert

On Mon, May 30, 2016 at 2:29 PM, Palle <pa...@sport.dk> wrote:

> Hi Robert.
>
> Thank you for the answer.
>
> I am looking at a rate of max 10.000 elements / 10 seconds, so
> Elastic/Kibana is probably the way to go. I'll find a way to model it.
>
> Thanks.
>
> /Palle
>
> ----- Original meddelelse -----
>
> *Fra:* Robert Metzger <rm...@apache.org>
> *Til:* user@flink.apache.org <us...@flink.apache.org>
> *Dato:* Man, 30. maj 2016 12:31
>
> *Emne:* Re: Visualize result of Flink job
>
> Hi Palle,
> I think there is currently no way of sending the data from a streaming
> Flink job into Zeppelin.
> What rate / amount of data do you expect to send every 10 seconds to the
> visualization tool?
> People have used Flink -> ES -> Kibana for this purpose in the past [1],
> but I think you can not send millions of records per second into ES.
> Something like 1000 - 5000 elements / second should easily work for a small
> ES setup.
> [1]
> https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana
>
> On Mon, May 30, 2016 at 10:00 AM, Palle <pa...@sport.dk> wrote:
>
>> OK, I found product that seems to be what I am looking for: Apache
>> Zeppelin. I will have a look into that one. If anyone can point me to an
>> example (Git) outputting data from Flink to the Zeppelin Notebook I would
>> be happy.
>>
>>
>> ----- Original meddelelse -----
>> > Fra: Palle <pa...@sport.dk>
>> > Til: user@flink.apache.org
>> > Dato: Man, 30. maj 2016 08:20
>> > Emne: Re: Visualize result of Flink job
>> >
>> > I know exactly what to visualize. As I wrote, it is the latest
>> > result of the Flink job I would like to visualize. There is no need
>> > to use elastic to find it first.
>> >
>> > The data I have is of such a  nature that  they every 10 seconds
>> > could be written into a file, meaning that the file at all times
>> > would contain the most recent results (at latest 10 seconds old). I
>> > am not interested in the history, and therefore I should think
>> > elastic is not the best fit. So my question is if anyone knows of a
>> > component (Apache or other) that can make the visualization a little
>> > nicer than just the file :-)
>> >
>> >
>> > ----- Original meddelelse -----
>> > > Fra: Kanstantsin Kamkou <kk...@gmail.com>
>> > > Til: user@flink.apache.org
>> > > Dato: Søn, 29. maj 2016 22:42
>> > > Emne: Re: Visualize result of Flink job
>> > >
>> > > >  I am thinking it may not be the best fit, because Elastic is by
>> > > nature a search engine that is good for trending and stuff like
>> > that
>> > > - not entire replacement of the current view.
>> > > Why u think that the elasticsearch is not the right tool? To
>> > > visualise
>> > > something u have to find what to visualise  first, right?
>> > >
>> > >
>> > > On Sun, May 29, 2016 at 8:20 PM, Palle <pa...@sport.dk> wrote:
>> > > > Hi there
>> > > >
>> > > > I am using Flink to analyse a lot of incoming data. Every 10
>> > > seconds it makes sense to present the analysis so far as some form
>> > > of visualization. Every 10 seconds I therefore will replace the
>> > > current contents of the visualization/presentation with the
>> > analysis
>> > > result of the most recent 10 seconds.
>> > > >
>> > > > I was first thinking of using ElasticSearch/Kibana for this
>> > > because I know it should be easy to set up, but I am thinking it
>> > may
>> > > not be the best fit, because Elastic is by nature a search engine
>> > > that is good for trending and stuff like that - not entire
>> > > replacement of the current view. And therefore I may also
>> > experience
>> > > difficulties implementing the view in Elastic.
>> > > >
>> > > > Does anyone know of any other visualization tools that work well
>> > > with Flink? ...where it is easy to export the result of a Flink
>> > job
>> > > to a user interface (web).
>> > > >
>> > > > Thanks
>> > > > Palle
>> >
>>
>
>
>

Re: Visualize result of Flink job

Posted by Palle <pa...@sport.dk>.
Hi Robert.

Thank you for the answer.

I am looking at a rate of max 10.000 elements / 10 seconds, so
Elastic/Kibana is probably the way to go. I'll find a way to model it.

Thanks.

/Palle

----- Original meddelelse -----

> Fra: Robert Metzger <rm...@apache.org>
> Til: user@flink.apache.org <us...@flink.apache.org>
> Dato: Man, 30. maj 2016 12:31
> Emne: Re: Visualize result of Flink job
> 
> Hi Palle, I think there is currently no way of sending the data from
> a streaming Flink job into Zeppelin.What rate / amount of data do you
> expect to send every 10 seconds to the visualization tool?People have
> used Flink -> ES -> Kibana for this purpose in the past [1], but I
> think you can not send millions of records per second into ES.
> Something like 1000 - 5000 elements / second should easily work for a
> small ES setup.[1]
> https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana [https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana]
> On Mon, May 30, 2016 at 10:00 AM, Palle < palle@sport.dk [mailto:palle@sport.dk]>
> wrote:
> 
>   OK, I found product that seems to be what I am looking for:
>   Apache Zeppelin. I will have a look into that one. If anyone can
>   point me to an example (Git) outputting data from Flink to the
>   Zeppelin Notebook I would be happy.
> 
> 
>   ----- Original meddelelse -----
>   > Fra: Palle < palle@sport.dk [mailto:palle@sport.dk]>
>   > Til: user@flink.apache.org [mailto:user@flink.apache.org]
>   > Dato: Man, 30. maj 2016 08:20
>   > Emne: Re: Visualize result of Flink job
>   >
>   > I know exactly what to visualize. As I wrote, it is the latest
>   > result of the Flink job I would like to visualize. There is no
>   need
>   > to use elastic to find it first.
>   >
>   > The data I have is of such a nature that they every 10 seconds
>   > could be written into a file, meaning that the file at all
>   times
>   > would contain the most recent results (at latest 10 seconds
>   old). I
>   > am not interested in the history, and therefore I should think
>   > elastic is not the best fit. So my question is if anyone knows
>   of a
>   > component (Apache or other) that can make the visualization a
>   little
>   > nicer than just the file :-)
>   >
>   >
>   > ----- Original meddelelse -----
>   > > Fra: Kanstantsin Kamkou < kkamkou@gmail.com [mailto:kkamkou@gmail.com]>
>   > > Til: user@flink.apache.org [mailto:user@flink.apache.org]
>   > > Dato: Søn, 29. maj 2016 22:42
>   > > Emne: Re: Visualize result of Flink job
>   > >
>   > > > I am thinking it may not be the best fit, because Elastic
>   is by
>   > > nature a search engine that is good for trending and stuff
>   like
>   > that
>   > > - not entire replacement of the current view.
>   > > Why u think that the elasticsearch is not the right tool? To
>   > > visualise
>   > > something u have to find what to visualise first, right?
>   > >
>   > >
>   > > On Sun, May 29, 2016 at 8:20 PM, Palle < palle@sport.dk [mailto:palle@sport.dk]>
>   wrote:
>   > > > Hi there
>   > > >
>   > > > I am using Flink to analyse a lot of incoming data. Every
>   10
>   > > seconds it makes sense to present the analysis so far as some
>   form
>   > > of visualization. Every 10 seconds I therefore will replace
>   the
>   > > current contents of the visualization/presentation with the
>   > analysis
>   > > result of the most recent 10 seconds.
>   > > >
>   > > > I was first thinking of using ElasticSearch/Kibana for this
>   > > because I know it should be easy to set up, but I am thinking
>   it
>   > may
>   > > not be the best fit, because Elastic is by nature a search
>   engine
>   > > that is good for trending and stuff like that - not entire
>   > > replacement of the current view. And therefore I may also
>   > experience
>   > > difficulties implementing the view in Elastic.
>   > > >
>   > > > Does anyone know of any other visualization tools that work
>   well
>   > > with Flink? ...where it is easy to export the result of a
>   Flink
>   > job
>   > > to a user interface (web).
>   > > >
>   > > > Thanks
>   > > > Palle
>   >


Re: Visualize result of Flink job

Posted by Robert Metzger <rm...@apache.org>.
Hi Palle,
I think there is currently no way of sending the data from a streaming
Flink job into Zeppelin.
What rate / amount of data do you expect to send every 10 seconds to the
visualization tool?

People have used Flink -> ES -> Kibana for this purpose in the past [1],
but I think you can not send millions of records per second into ES.
Something like 1000 - 5000 elements / second should easily work for a small
ES setup.


[1]
https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana

On Mon, May 30, 2016 at 10:00 AM, Palle <pa...@sport.dk> wrote:

> OK, I found product that seems to be what I am looking for: Apache
> Zeppelin. I will have a look into that one. If anyone can point me to an
> example (Git) outputting data from Flink to the Zeppelin Notebook I would
> be happy.
>
>
> ----- Original meddelelse -----
> > Fra: Palle <pa...@sport.dk>
> > Til: user@flink.apache.org
> > Dato: Man, 30. maj 2016 08:20
> > Emne: Re: Visualize result of Flink job
> >
> > I know exactly what to visualize. As I wrote, it is the latest
> > result of the Flink job I would like to visualize. There is no need
> > to use elastic to find it first.
> >
> > The data I have is of such a  nature that  they every 10 seconds
> > could be written into a file, meaning that the file at all times
> > would contain the most recent results (at latest 10 seconds old). I
> > am not interested in the history, and therefore I should think
> > elastic is not the best fit. So my question is if anyone knows of a
> > component (Apache or other) that can make the visualization a little
> > nicer than just the file :-)
> >
> >
> > ----- Original meddelelse -----
> > > Fra: Kanstantsin Kamkou <kk...@gmail.com>
> > > Til: user@flink.apache.org
> > > Dato: Søn, 29. maj 2016 22:42
> > > Emne: Re: Visualize result of Flink job
> > >
> > > >  I am thinking it may not be the best fit, because Elastic is by
> > > nature a search engine that is good for trending and stuff like
> > that
> > > - not entire replacement of the current view.
> > > Why u think that the elasticsearch is not the right tool? To
> > > visualise
> > > something u have to find what to visualise  first, right?
> > >
> > >
> > > On Sun, May 29, 2016 at 8:20 PM, Palle <pa...@sport.dk> wrote:
> > > > Hi there
> > > >
> > > > I am using Flink to analyse a lot of incoming data. Every 10
> > > seconds it makes sense to present the analysis so far as some form
> > > of visualization. Every 10 seconds I therefore will replace the
> > > current contents of the visualization/presentation with the
> > analysis
> > > result of the most recent 10 seconds.
> > > >
> > > > I was first thinking of using ElasticSearch/Kibana for this
> > > because I know it should be easy to set up, but I am thinking it
> > may
> > > not be the best fit, because Elastic is by nature a search engine
> > > that is good for trending and stuff like that - not entire
> > > replacement of the current view. And therefore I may also
> > experience
> > > difficulties implementing the view in Elastic.
> > > >
> > > > Does anyone know of any other visualization tools that work well
> > > with Flink? ...where it is easy to export the result of a Flink
> > job
> > > to a user interface (web).
> > > >
> > > > Thanks
> > > > Palle
> >
>
>

Re: Visualize result of Flink job

Posted by Palle <pa...@sport.dk>.
OK, I found product that seems to be what I am looking for: Apache Zeppelin. I will have a look into that one. If anyone can point me to an example (Git) outputting data from Flink to the Zeppelin Notebook I would be happy.


----- Original meddelelse -----
> Fra: Palle <pa...@sport.dk>
> Til: user@flink.apache.org
> Dato: Man, 30. maj 2016 08:20
> Emne: Re: Visualize result of Flink job
> 
> I know exactly what to visualize. As I wrote, it is the latest
> result of the Flink job I would like to visualize. There is no need
> to use elastic to find it first.
> 
> The data I have is of such a  nature that  they every 10 seconds
> could be written into a file, meaning that the file at all times
> would contain the most recent results (at latest 10 seconds old). I
> am not interested in the history, and therefore I should think
> elastic is not the best fit. So my question is if anyone knows of a
> component (Apache or other) that can make the visualization a little
> nicer than just the file :-)
> 
> 
> ----- Original meddelelse -----
> > Fra: Kanstantsin Kamkou <kk...@gmail.com>
> > Til: user@flink.apache.org
> > Dato: Søn, 29. maj 2016 22:42
> > Emne: Re: Visualize result of Flink job
> > 
> > >  I am thinking it may not be the best fit, because Elastic is by
> > nature a search engine that is good for trending and stuff like
> that
> > - not entire replacement of the current view.
> > Why u think that the elasticsearch is not the right tool? To
> > visualise
> > something u have to find what to visualise  first, right?
> > 
> > 
> > On Sun, May 29, 2016 at 8:20 PM, Palle <pa...@sport.dk> wrote:
> > > Hi there
> > >
> > > I am using Flink to analyse a lot of incoming data. Every 10
> > seconds it makes sense to present the analysis so far as some form
> > of visualization. Every 10 seconds I therefore will replace the
> > current contents of the visualization/presentation with the
> analysis
> > result of the most recent 10 seconds.
> > >
> > > I was first thinking of using ElasticSearch/Kibana for this
> > because I know it should be easy to set up, but I am thinking it
> may
> > not be the best fit, because Elastic is by nature a search engine
> > that is good for trending and stuff like that - not entire
> > replacement of the current view. And therefore I may also
> experience
> > difficulties implementing the view in Elastic.
> > >
> > > Does anyone know of any other visualization tools that work well
> > with Flink? ...where it is easy to export the result of a Flink
> job
> > to a user interface (web).
> > >
> > > Thanks
> > > Palle
> 


Re: Visualize result of Flink job

Posted by Palle <pa...@sport.dk>.
I know exactly what to visualize. As I wrote, it is the latest result of the Flink job I would like to visualize. There is no need to use elastic to find it first.

The data I have is of such a  nature that  they every 10 seconds could be written into a file, meaning that the file at all times would contain the most recent results (at latest 10 seconds old). I am not interested in the history, and therefore I should think elastic is not the best fit. So my question is if anyone knows of a component (Apache or other) that can make the visualization a little nicer than just the file :-)


----- Original meddelelse -----
> Fra: Kanstantsin Kamkou <kk...@gmail.com>
> Til: user@flink.apache.org
> Dato: Søn, 29. maj 2016 22:42
> Emne: Re: Visualize result of Flink job
> 
> >  I am thinking it may not be the best fit, because Elastic is by
> nature a search engine that is good for trending and stuff like that
> - not entire replacement of the current view.
> Why u think that the elasticsearch is not the right tool? To
> visualise
> something u have to find what to visualise  first, right?
> 
> 
> On Sun, May 29, 2016 at 8:20 PM, Palle <pa...@sport.dk> wrote:
> > Hi there
> >
> > I am using Flink to analyse a lot of incoming data. Every 10
> seconds it makes sense to present the analysis so far as some form
> of visualization. Every 10 seconds I therefore will replace the
> current contents of the visualization/presentation with the analysis
> result of the most recent 10 seconds.
> >
> > I was first thinking of using ElasticSearch/Kibana for this
> because I know it should be easy to set up, but I am thinking it may
> not be the best fit, because Elastic is by nature a search engine
> that is good for trending and stuff like that - not entire
> replacement of the current view. And therefore I may also experience
> difficulties implementing the view in Elastic.
> >
> > Does anyone know of any other visualization tools that work well
> with Flink? ...where it is easy to export the result of a Flink job
> to a user interface (web).
> >
> > Thanks
> > Palle


Re: Visualize result of Flink job

Posted by Kanstantsin Kamkou <kk...@gmail.com>.
>  I am thinking it may not be the best fit, because Elastic is by nature a search engine that is good for trending and stuff like that - not entire replacement of the current view.
Why u think that the elasticsearch is not the right tool? To visualise
something u have to find what to visualise  first, right?


On Sun, May 29, 2016 at 8:20 PM, Palle <pa...@sport.dk> wrote:
> Hi there
>
> I am using Flink to analyse a lot of incoming data. Every 10 seconds it makes sense to present the analysis so far as some form of visualization. Every 10 seconds I therefore will replace the current contents of the visualization/presentation with the analysis result of the most recent 10 seconds.
>
> I was first thinking of using ElasticSearch/Kibana for this because I know it should be easy to set up, but I am thinking it may not be the best fit, because Elastic is by nature a search engine that is good for trending and stuff like that - not entire replacement of the current view. And therefore I may also experience difficulties implementing the view in Elastic.
>
> Does anyone know of any other visualization tools that work well with Flink? ...where it is easy to export the result of a Flink job to a user interface (web).
>
> Thanks
> Palle