You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Romain Manni-Bucau <rm...@gmail.com> on 2017/05/25 17:48:07 UTC
graph generator?
Hello guys,
does beam have a graph generator from a pipeline? Not sure current API
fully allows to bypass the runner to just get the beam graph but it can
help to have a small main generating a png/svg/ascii/ditaa (or a
maven/gradle plugin ;))
Needed that in by hazelcast-jet work to visualize the graph, i have a quick
and dirty impl based on jung (BSD license :() based on jet graph but think
it should be pretty trivial to use directly beam graph based on a pipeline
visitor.
Mainly sending this mail to share it in case anyone needs it more than
anything else:
https://gist.github.com/rmannibucau/b5f4e310b40ce414f95f6e22530bbe6e
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://blog-rmannibucau.rhcloud.com> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
<https://javaeefactory-rmannibucau.rhcloud.com>
Re: graph generator?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
agree, it's what I had in mind with "some tools".
Regards
JB
On 05/25/2017 08:06 PM, Dan Halperin wrote:
> I think that a util that converted from the Runner API definition of a
> pipeline into some sort of graph format (like DOT?) would be generally
> useful. By using the Runner API, the tool would be SDK- and
> Runner-independent view of the pipeline.
>
> On Thu, May 25, 2017 at 10:54 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi,
>>
>> If you mean a graphical tool, no, it's up to each execution engine (it's
>> what we showed last week at ApacheCon with Davor).
>>
>> Some tools can graphically generate the graph with the corresponding Beam
>> pipeline.
>>
>> Regards
>> JB
>>
>>
>> On 05/25/2017 07:48 PM, Romain Manni-Bucau wrote:
>>
>>> Hello guys,
>>>
>>> does beam have a graph generator from a pipeline? Not sure current API
>>> fully allows to bypass the runner to just get the beam graph but it can
>>> help to have a small main generating a png/svg/ascii/ditaa (or a
>>> maven/gradle plugin ;))
>>>
>>> Needed that in by hazelcast-jet work to visualize the graph, i have a
>>> quick
>>> and dirty impl based on jung (BSD license :() based on jet graph but think
>>> it should be pretty trivial to use directly beam graph based on a pipeline
>>> visitor.
>>>
>>> Mainly sending this mail to share it in case anyone needs it more than
>>> anything else:
>>> https://gist.github.com/rmannibucau/b5f4e310b40ce414f95f6e22530bbe6e
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://blog-rmannibucau.rhcloud.com> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github <
>>> https://github.com/rmannibucau> |
>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
>>> <https://javaeefactory-rmannibucau.rhcloud.com>
>>>
>>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: graph generator?
Posted by Sourabh Bajaj <so...@google.com.INVALID>.
+1
On Thu, May 25, 2017 at 11:10 AM Lukasz Cwik <lc...@google.com.invalid>
wrote:
> +1 on Runner API to dot format or plantuml format.
>
> On Thu, May 25, 2017 at 11:06 AM, Dan Halperin <dhalperi@google.com.invalid
> >
> wrote:
>
> > I think that a util that converted from the Runner API definition of a
> > pipeline into some sort of graph format (like DOT?) would be generally
> > useful. By using the Runner API, the tool would be SDK- and
> > Runner-independent view of the pipeline.
> >
> > On Thu, May 25, 2017 at 10:54 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi,
> > >
> > > If you mean a graphical tool, no, it's up to each execution engine
> (it's
> > > what we showed last week at ApacheCon with Davor).
> > >
> > > Some tools can graphically generate the graph with the corresponding
> Beam
> > > pipeline.
> > >
> > > Regards
> > > JB
> > >
> > >
> > > On 05/25/2017 07:48 PM, Romain Manni-Bucau wrote:
> > >
> > >> Hello guys,
> > >>
> > >> does beam have a graph generator from a pipeline? Not sure current API
> > >> fully allows to bypass the runner to just get the beam graph but it
> can
> > >> help to have a small main generating a png/svg/ascii/ditaa (or a
> > >> maven/gradle plugin ;))
> > >>
> > >> Needed that in by hazelcast-jet work to visualize the graph, i have a
> > >> quick
> > >> and dirty impl based on jung (BSD license :() based on jet graph but
> > think
> > >> it should be pretty trivial to use directly beam graph based on a
> > pipeline
> > >> visitor.
> > >>
> > >> Mainly sending this mail to share it in case anyone needs it more than
> > >> anything else:
> > >> https://gist.github.com/rmannibucau/b5f4e310b40ce414f95f6e22530bbe6e
> > >>
> > >> Romain Manni-Bucau
> > >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > >> <https://blog-rmannibucau.rhcloud.com> | Old Blog
> > >> <http://rmannibucau.wordpress.com> | Github <
> > >> https://github.com/rmannibucau> |
> > >> LinkedIn <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
> > >> <https://javaeefactory-rmannibucau.rhcloud.com>
> > >>
> > >>
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
>
Re: graph generator?
Posted by Lukasz Cwik <lc...@google.com.INVALID>.
+1 on Runner API to dot format or plantuml format.
On Thu, May 25, 2017 at 11:06 AM, Dan Halperin <dh...@google.com.invalid>
wrote:
> I think that a util that converted from the Runner API definition of a
> pipeline into some sort of graph format (like DOT?) would be generally
> useful. By using the Runner API, the tool would be SDK- and
> Runner-independent view of the pipeline.
>
> On Thu, May 25, 2017 at 10:54 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi,
> >
> > If you mean a graphical tool, no, it's up to each execution engine (it's
> > what we showed last week at ApacheCon with Davor).
> >
> > Some tools can graphically generate the graph with the corresponding Beam
> > pipeline.
> >
> > Regards
> > JB
> >
> >
> > On 05/25/2017 07:48 PM, Romain Manni-Bucau wrote:
> >
> >> Hello guys,
> >>
> >> does beam have a graph generator from a pipeline? Not sure current API
> >> fully allows to bypass the runner to just get the beam graph but it can
> >> help to have a small main generating a png/svg/ascii/ditaa (or a
> >> maven/gradle plugin ;))
> >>
> >> Needed that in by hazelcast-jet work to visualize the graph, i have a
> >> quick
> >> and dirty impl based on jung (BSD license :() based on jet graph but
> think
> >> it should be pretty trivial to use directly beam graph based on a
> pipeline
> >> visitor.
> >>
> >> Mainly sending this mail to share it in case anyone needs it more than
> >> anything else:
> >> https://gist.github.com/rmannibucau/b5f4e310b40ce414f95f6e22530bbe6e
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >> <https://blog-rmannibucau.rhcloud.com> | Old Blog
> >> <http://rmannibucau.wordpress.com> | Github <
> >> https://github.com/rmannibucau> |
> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
> >> <https://javaeefactory-rmannibucau.rhcloud.com>
> >>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
Re: graph generator?
Posted by Dan Halperin <dh...@google.com.INVALID>.
I think that a util that converted from the Runner API definition of a
pipeline into some sort of graph format (like DOT?) would be generally
useful. By using the Runner API, the tool would be SDK- and
Runner-independent view of the pipeline.
On Thu, May 25, 2017 at 10:54 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi,
>
> If you mean a graphical tool, no, it's up to each execution engine (it's
> what we showed last week at ApacheCon with Davor).
>
> Some tools can graphically generate the graph with the corresponding Beam
> pipeline.
>
> Regards
> JB
>
>
> On 05/25/2017 07:48 PM, Romain Manni-Bucau wrote:
>
>> Hello guys,
>>
>> does beam have a graph generator from a pipeline? Not sure current API
>> fully allows to bypass the runner to just get the beam graph but it can
>> help to have a small main generating a png/svg/ascii/ditaa (or a
>> maven/gradle plugin ;))
>>
>> Needed that in by hazelcast-jet work to visualize the graph, i have a
>> quick
>> and dirty impl based on jung (BSD license :() based on jet graph but think
>> it should be pretty trivial to use directly beam graph based on a pipeline
>> visitor.
>>
>> Mainly sending this mail to share it in case anyone needs it more than
>> anything else:
>> https://gist.github.com/rmannibucau/b5f4e310b40ce414f95f6e22530bbe6e
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://blog-rmannibucau.rhcloud.com> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github <
>> https://github.com/rmannibucau> |
>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
>> <https://javaeefactory-rmannibucau.rhcloud.com>
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: graph generator?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
If you mean a graphical tool, no, it's up to each execution engine (it's what we
showed last week at ApacheCon with Davor).
Some tools can graphically generate the graph with the corresponding Beam pipeline.
Regards
JB
On 05/25/2017 07:48 PM, Romain Manni-Bucau wrote:
> Hello guys,
>
> does beam have a graph generator from a pipeline? Not sure current API
> fully allows to bypass the runner to just get the beam graph but it can
> help to have a small main generating a png/svg/ascii/ditaa (or a
> maven/gradle plugin ;))
>
> Needed that in by hazelcast-jet work to visualize the graph, i have a quick
> and dirty impl based on jung (BSD license :() based on jet graph but think
> it should be pretty trivial to use directly beam graph based on a pipeline
> visitor.
>
> Mainly sending this mail to share it in case anyone needs it more than
> anything else:
> https://gist.github.com/rmannibucau/b5f4e310b40ce414f95f6e22530bbe6e
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://blog-rmannibucau.rhcloud.com> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
> <https://javaeefactory-rmannibucau.rhcloud.com>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com