You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ken Krugler <kk...@transpac.com> on 2017/02/21 22:59:16 UTC
Visualizing topologies
Hi list,
I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools)
So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
The results look like this...
http://scaleunlimited.com/downloads/CrawlTopology.png
If this is useful (and doesn’t already exist) then I can easily drop it in somewhere.
— Ken
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
Re: Visualizing topologies
Posted by Ken Krugler <kk...@transpac.com>.
Hi Pat,
> On Feb 21, 2017, at 6:01pm, Pattarawat Chormai <pa...@gmail.com> wrote:
>
> Hi Ken,
>
> Maybe you can look into this one : http://flink.apache.org/visualizer/.
Thanks, that’s interesting and convenient.
Though I’d probably keep using OmniGraffle with a dot file as that gives me the ability to edit/annotate as needed.
Thoughts on having the iteration sink connect to the iteration source? I found that helpful when working on my topology.
Regards,
— Ken
>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <kk...@transpac.com> wrote:
>>
>> Hi list,
>>
>> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools)
>>
>> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
>>
>> The results look like this...
>>
>> http://scaleunlimited.com/downloads/CrawlTopology.png
>>
>> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere.
>>
>> — Ken
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
Re: Visualizing topologies
Posted by Ken Krugler <kk...@transpac.com>.
Hi Greg,
The use case is to create a visualization of the topology.
So I don’t think there’s any reason to “act on the dot file from within the user program”
Regards,
— Ken
> On Feb 24, 2017, at 7:51am, Greg Hogan <co...@greghogan.com> wrote:
>
> Ken and Fabian,
>
> Is the use case to generate and act on the dot file from within the user
> program? Would it be more maintainable to make the plan JSON more
> accessible (through the CLI and web interface) which users could then pipe
> through a converter script?
>
> Greg
>
> On Fri, Feb 24, 2017 at 4:55 AM, Fabian Hueske <fh...@gmail.com> wrote:
>
>> Hi Ken,
>>
>> I think this would be an interesting feature!
>> I'd suggest to open a JIRA for it.
>>
>> When extending the API of core classes such as ExecutionEnvironment, there
>> is often some discussion whether the feature is important enough or whether
>> it should be rather added to some external util class (which makes it
>> obviously less visible).
>> I think this feature would be valuable enough to be added directly into the
>> ExecutionEnvironment. DOT is a common format for graph and flow
>> visualizations.
>>
>> Let's see what others think.
>>
>> Best, Fabian
>>
>>
>> 2017-02-24 0:15 GMT+01:00 Ken Krugler <kk...@transpac.com>:
>>
>>> Hi Ufuk,
>>>
>>>> On Feb 22, 2017, at 2:18am, Ufuk Celebi <uc...@apache.org> wrote:
>>>>
>>>> Hey Ken!
>>>>
>>>> This looks really good. +1 to make this available publicly.
>>>>
>>>> We can link it from the Flink website and the viz tool Pat linked to.
>>>> The vizualizer has currently some open issues, it is not up to date
>>>> with the one that is part of the Flink web UI.
>>>
>>> I wasn’t planning on running a web interface to the tool.
>>>
>>> My thought was to just add a writeDotFile(“path to file") method to the
>>> StreamExecutionEnvironment…this is similar to what Cascading supports.
>>>
>>> If that’s reasonable, I’ll open an issue and attach the code.
>>>
>>> — Ken
>>>
>>>
>>>> On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai
>>>> <pa...@gmail.com> wrote:
>>>>> Hi Ken,
>>>>>
>>>>> Maybe you can look into this one : http://flink.apache.org/
>> visualizer/.
>>>>>
>>>>> - Pat
>>>>>
>>>>>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <
>> kkrugler_lists@transpac.com>
>>> wrote:
>>>>>>
>>>>>> Hi list,
>>>>>>
>>>>>> I poked around a bit and didn’t see a way to easily visualize
>>> workflows in the same way I was used to with Cascading (generate a
>> blah.dot
>>> file, open with OminGraffle or other graph tools)
>>>>>>
>>>>>> So I wrote a bit of code that takes the JSON output from
>>> StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
>>>>>>
>>>>>> The results look like this...
>>>>>>
>>>>>> http://scaleunlimited.com/downloads/CrawlTopology.png
>>>>>>
>>>>>> If this is useful (and doesn’t already exist) then I can easily drop
>>> it in somewhere.
>>>>>>
>>>>>> — Ken
>>>
>>> --------------------------
>>> Ken Krugler
>>> +1 530-210-6378
>>> http://www.scaleunlimited.com
>>> custom big data solutions & training
>>> Hadoop, Cascading, Cassandra & Solr
>>>
>>>
>>>
>>>
>>
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
Re: Visualizing topologies
Posted by Greg Hogan <co...@greghogan.com>.
Ken and Fabian,
Is the use case to generate and act on the dot file from within the user
program? Would it be more maintainable to make the plan JSON more
accessible (through the CLI and web interface) which users could then pipe
through a converter script?
Greg
On Fri, Feb 24, 2017 at 4:55 AM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi Ken,
>
> I think this would be an interesting feature!
> I'd suggest to open a JIRA for it.
>
> When extending the API of core classes such as ExecutionEnvironment, there
> is often some discussion whether the feature is important enough or whether
> it should be rather added to some external util class (which makes it
> obviously less visible).
> I think this feature would be valuable enough to be added directly into the
> ExecutionEnvironment. DOT is a common format for graph and flow
> visualizations.
>
> Let's see what others think.
>
> Best, Fabian
>
>
> 2017-02-24 0:15 GMT+01:00 Ken Krugler <kk...@transpac.com>:
>
> > Hi Ufuk,
> >
> > > On Feb 22, 2017, at 2:18am, Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > Hey Ken!
> > >
> > > This looks really good. +1 to make this available publicly.
> > >
> > > We can link it from the Flink website and the viz tool Pat linked to.
> > > The vizualizer has currently some open issues, it is not up to date
> > > with the one that is part of the Flink web UI.
> >
> > I wasn’t planning on running a web interface to the tool.
> >
> > My thought was to just add a writeDotFile(“path to file") method to the
> > StreamExecutionEnvironment…this is similar to what Cascading supports.
> >
> > If that’s reasonable, I’ll open an issue and attach the code.
> >
> > — Ken
> >
> >
> > > On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai
> > > <pa...@gmail.com> wrote:
> > >> Hi Ken,
> > >>
> > >> Maybe you can look into this one : http://flink.apache.org/
> visualizer/.
> > >>
> > >> - Pat
> > >>
> > >>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <
> kkrugler_lists@transpac.com>
> > wrote:
> > >>>
> > >>> Hi list,
> > >>>
> > >>> I poked around a bit and didn’t see a way to easily visualize
> > workflows in the same way I was used to with Cascading (generate a
> blah.dot
> > file, open with OminGraffle or other graph tools)
> > >>>
> > >>> So I wrote a bit of code that takes the JSON output from
> > StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
> > >>>
> > >>> The results look like this...
> > >>>
> > >>> http://scaleunlimited.com/downloads/CrawlTopology.png
> > >>>
> > >>> If this is useful (and doesn’t already exist) then I can easily drop
> > it in somewhere.
> > >>>
> > >>> — Ken
> >
> > --------------------------
> > Ken Krugler
> > +1 530-210-6378
> > http://www.scaleunlimited.com
> > custom big data solutions & training
> > Hadoop, Cascading, Cassandra & Solr
> >
> >
> >
> >
>
Re: Visualizing topologies
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Ken,
I think this would be an interesting feature!
I'd suggest to open a JIRA for it.
When extending the API of core classes such as ExecutionEnvironment, there
is often some discussion whether the feature is important enough or whether
it should be rather added to some external util class (which makes it
obviously less visible).
I think this feature would be valuable enough to be added directly into the
ExecutionEnvironment. DOT is a common format for graph and flow
visualizations.
Let's see what others think.
Best, Fabian
2017-02-24 0:15 GMT+01:00 Ken Krugler <kk...@transpac.com>:
> Hi Ufuk,
>
> > On Feb 22, 2017, at 2:18am, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > Hey Ken!
> >
> > This looks really good. +1 to make this available publicly.
> >
> > We can link it from the Flink website and the viz tool Pat linked to.
> > The vizualizer has currently some open issues, it is not up to date
> > with the one that is part of the Flink web UI.
>
> I wasn’t planning on running a web interface to the tool.
>
> My thought was to just add a writeDotFile(“path to file") method to the
> StreamExecutionEnvironment…this is similar to what Cascading supports.
>
> If that’s reasonable, I’ll open an issue and attach the code.
>
> — Ken
>
>
> > On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai
> > <pa...@gmail.com> wrote:
> >> Hi Ken,
> >>
> >> Maybe you can look into this one : http://flink.apache.org/visualizer/.
> >>
> >> - Pat
> >>
> >>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <kk...@transpac.com>
> wrote:
> >>>
> >>> Hi list,
> >>>
> >>> I poked around a bit and didn’t see a way to easily visualize
> workflows in the same way I was used to with Cascading (generate a blah.dot
> file, open with OminGraffle or other graph tools)
> >>>
> >>> So I wrote a bit of code that takes the JSON output from
> StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
> >>>
> >>> The results look like this...
> >>>
> >>> http://scaleunlimited.com/downloads/CrawlTopology.png
> >>>
> >>> If this is useful (and doesn’t already exist) then I can easily drop
> it in somewhere.
> >>>
> >>> — Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
>
>
>
Re: Visualizing topologies
Posted by Ken Krugler <kk...@transpac.com>.
Hi Ufuk,
> On Feb 22, 2017, at 2:18am, Ufuk Celebi <uc...@apache.org> wrote:
>
> Hey Ken!
>
> This looks really good. +1 to make this available publicly.
>
> We can link it from the Flink website and the viz tool Pat linked to.
> The vizualizer has currently some open issues, it is not up to date
> with the one that is part of the Flink web UI.
I wasn’t planning on running a web interface to the tool.
My thought was to just add a writeDotFile(“path to file") method to the StreamExecutionEnvironment…this is similar to what Cascading supports.
If that’s reasonable, I’ll open an issue and attach the code.
— Ken
> On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai
> <pa...@gmail.com> wrote:
>> Hi Ken,
>>
>> Maybe you can look into this one : http://flink.apache.org/visualizer/.
>>
>> - Pat
>>
>>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <kk...@transpac.com> wrote:
>>>
>>> Hi list,
>>>
>>> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools)
>>>
>>> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
>>>
>>> The results look like this...
>>>
>>> http://scaleunlimited.com/downloads/CrawlTopology.png
>>>
>>> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere.
>>>
>>> — Ken
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
Re: Visualizing topologies
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Ken!
This looks really good. +1 to make this available publicly.
We can link it from the Flink website and the viz tool Pat linked to.
The vizualizer has currently some open issues, it is not up to date
with the one that is part of the Flink web UI.
– Ufuk
On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai
<pa...@gmail.com> wrote:
> Hi Ken,
>
> Maybe you can look into this one : http://flink.apache.org/visualizer/.
>
> - Pat
>
>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <kk...@transpac.com> wrote:
>>
>> Hi list,
>>
>> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools)
>>
>> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
>>
>> The results look like this...
>>
>> http://scaleunlimited.com/downloads/CrawlTopology.png
>>
>> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere.
>>
>> — Ken
>>
>> --------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://www.scaleunlimited.com
>> custom big data solutions & training
>> Hadoop, Cascading, Cassandra & Solr
>>
>>
>>
>
Re: Visualizing topologies
Posted by Pattarawat Chormai <pa...@gmail.com>.
Hi Ken,
Maybe you can look into this one : http://flink.apache.org/visualizer/.
- Pat
> On Feb 21, 2017, at 11:59 PM, Ken Krugler <kk...@transpac.com> wrote:
>
> Hi list,
>
> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools)
>
> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
>
> The results look like this...
>
> http://scaleunlimited.com/downloads/CrawlTopology.png
>
> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere.
>
> — Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
>
>