You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by camelia c <ca...@yahoo.com> on 2014/06/26 18:45:21 UTC
How to display DAG?
Hello,
This is my second question for today.
Suppose that I managed to create the DAG successfully, how can I display the plan (with vertices and edges) in a user-friendly way?
I found a nice method DAGUtils.generateSimpleJSONPlan(dagPlan)
but the dagPlan can be obtained using a class (DAGImpl) implementing the interface org.apache.tez.dag.app.DAG (the getJobPlan() method) , not an instance of org.apache.tez.dag.api.DAG.
.
Thank You very much in advance.
Yours sincerely,
Camelia
Re: How to display DAG?
Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
+1, we should move the generateViz to dag-api to allow user generate graph
without submitting it to yarn.
Best Regards,
Jeff Zhang
On Fri, Jun 27, 2014 at 1:27 AM, Hitesh Shah <hi...@apache.org> wrote:
> Hello Camelia
>
> There is an open jira - https://issues.apache.org/jira/browse/TEZ-974 -
> for the feature that you are requesting. You would need to potentially
> re-use the code that was written as part of TEZ-876. If you are willing to
> work on it, let us know. We can assign the jira to you.
>
> thanks
> — Hitesh
>
> On Jun 26, 2014, at 10:11 AM, Bikas Saha <bi...@hortonworks.com> wrote:
>
> > If you have the Tez source code you can look at
> DAGAppMaster.generateDAGVizFile() This walks at the DAGPlan object and
> generates this dot plot. The DAGPlan object is a protobuf based version of
> the DAG plan produced via the DAG API.
> >
> > You could potentially use the same code inside TezClient to generate the
> dot plot on the client side. If you end up doing this please open a jira
> and contribute the code back to the project. We do it on the app master
> because the final expansion of the graph happens there (along with other
> runtime modifications) and so that graph plot is more accurate.
> >
> > Bikas
> >
> > From: camelia c [mailto:camelie_1985@yahoo.com]
> > Sent: Thursday, June 26, 2014 9:59 AM
> > To: user@tez.incubator.apache.org
> > Subject: Re: How to display DAG?
> >
> >
> > I'd like to display it before launch in execution. Is there a method
> available that traverses the dag obtained with the DAG API and displays
> debug-like info, or should I write one myself?
> >
> > Thank You for Your prompt answer.
> >
> > Camelia
> >
> > From: Bikas Saha <bi...@hortonworks.com>
> > To: user@tez.incubator.apache.org; camelia c <ca...@yahoo.com>
> > Sent: Thursday, June 26, 2014 7:51 PM
> > Subject: RE: How to display DAG?
> >
> > That method is probably used to generate history events for recovery.
> >
> > When the dag runs, it generates a dot plot of the graph and writes it to
> its local dir. If you are running it locally then you can find it in the
> log dir of your application master container. If you are running on the
> cluster and have log aggregation turned on, then you can download it via
> the “yarn log” command. The file name would be like
> “dag_1403748209003_0006_1.dot”. Then you can view it using a dot plot
> viewing tool.
> >
> >
> > Bikas
> >
> > From: camelia c [mailto:camelie_1985@yahoo.com]
> > Sent: Thursday, June 26, 2014 9:45 AM
> > To: user@tez.incubator.apache.org
> > Subject: How to display DAG?
> >
> > Hello,
> >
> > This is my second question for today.
> > Suppose that I managed to create the DAG successfully, how can I display
> the plan (with vertices and edges) in a user-friendly way?
> >
> > I found a nice method DAGUtils.generateSimpleJSONPlan(dagPlan)
> >
> > but the dagPlan can be obtained using a class (DAGImpl) implementing the
> interface org.apache.tez.dag.app.DAG (the getJobPlan() method) , not
> an instance of org.apache.tez.dag.api.DAG.
> > .
> >
> > Thank You very much in advance.
> >
> > Yours sincerely,
> > Camelia
> >
> >
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
> >
> >
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: How to display DAG?
Posted by Hitesh Shah <hi...@apache.org>.
Hello Camelia
There is an open jira - https://issues.apache.org/jira/browse/TEZ-974 - for the feature that you are requesting. You would need to potentially re-use the code that was written as part of TEZ-876. If you are willing to work on it, let us know. We can assign the jira to you.
thanks
— Hitesh
On Jun 26, 2014, at 10:11 AM, Bikas Saha <bi...@hortonworks.com> wrote:
> If you have the Tez source code you can look at DAGAppMaster.generateDAGVizFile() This walks at the DAGPlan object and generates this dot plot. The DAGPlan object is a protobuf based version of the DAG plan produced via the DAG API.
>
> You could potentially use the same code inside TezClient to generate the dot plot on the client side. If you end up doing this please open a jira and contribute the code back to the project. We do it on the app master because the final expansion of the graph happens there (along with other runtime modifications) and so that graph plot is more accurate.
>
> Bikas
>
> From: camelia c [mailto:camelie_1985@yahoo.com]
> Sent: Thursday, June 26, 2014 9:59 AM
> To: user@tez.incubator.apache.org
> Subject: Re: How to display DAG?
>
>
> I'd like to display it before launch in execution. Is there a method available that traverses the dag obtained with the DAG API and displays debug-like info, or should I write one myself?
>
> Thank You for Your prompt answer.
>
> Camelia
>
> From: Bikas Saha <bi...@hortonworks.com>
> To: user@tez.incubator.apache.org; camelia c <ca...@yahoo.com>
> Sent: Thursday, June 26, 2014 7:51 PM
> Subject: RE: How to display DAG?
>
> That method is probably used to generate history events for recovery.
>
> When the dag runs, it generates a dot plot of the graph and writes it to its local dir. If you are running it locally then you can find it in the log dir of your application master container. If you are running on the cluster and have log aggregation turned on, then you can download it via the “yarn log” command. The file name would be like “dag_1403748209003_0006_1.dot”. Then you can view it using a dot plot viewing tool.
>
>
> Bikas
>
> From: camelia c [mailto:camelie_1985@yahoo.com]
> Sent: Thursday, June 26, 2014 9:45 AM
> To: user@tez.incubator.apache.org
> Subject: How to display DAG?
>
> Hello,
>
> This is my second question for today.
> Suppose that I managed to create the DAG successfully, how can I display the plan (with vertices and edges) in a user-friendly way?
>
> I found a nice method DAGUtils.generateSimpleJSONPlan(dagPlan)
>
> but the dagPlan can be obtained using a class (DAGImpl) implementing the interface org.apache.tez.dag.app.DAG (the getJobPlan() method) , not an instance of org.apache.tez.dag.api.DAG.
> .
>
> Thank You very much in advance.
>
> Yours sincerely,
> Camelia
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: How to display DAG?
Posted by Bikas Saha <bi...@hortonworks.com>.
If you have the Tez source code you can look at
DAGAppMaster.generateDAGVizFile() This walks at the DAGPlan object and
generates this dot plot. The DAGPlan object is a protobuf based version of
the DAG plan produced via the DAG API.
You could potentially use the same code inside TezClient to generate the
dot plot on the client side. If you end up doing this please open a jira
and contribute the code back to the project. We do it on the app master
because the final expansion of the graph happens there (along with other
runtime modifications) and so that graph plot is more accurate.
Bikas
*From:* camelia c [mailto:camelie_1985@yahoo.com]
*Sent:* Thursday, June 26, 2014 9:59 AM
*To:* user@tez.incubator.apache.org
*Subject:* Re: How to display DAG?
I'd like to display it before launch in execution. Is there a method
available that traverses the dag obtained with the DAG API and displays
debug-like info, or should I write one myself?
Thank You for Your prompt answer.
Camelia
------------------------------
*From:* Bikas Saha <bi...@hortonworks.com>
*To:* user@tez.incubator.apache.org; camelia c <ca...@yahoo.com>
*Sent:* Thursday, June 26, 2014 7:51 PM
*Subject:* RE: How to display DAG?
That method is probably used to generate history events for recovery.
When the dag runs, it generates a dot plot of the graph and writes it to
its local dir. If you are running it locally then you can find it in the
log dir of your application master container. If you are running on the
cluster and have log aggregation turned on, then you can download it via
the “yarn log” command. The file name would be like
“dag_1403748209003_0006_1.dot”. Then you can view it using a dot plot
viewing tool.
Bikas
*From:* camelia c [mailto:camelie_1985@yahoo.com]
*Sent:* Thursday, June 26, 2014 9:45 AM
*To:* user@tez.incubator.apache.org
*Subject:* How to display DAG?
Hello,
This is my second question for today.
Suppose that I managed to create the DAG successfully, how can I display
the plan (with vertices and edges) in a user-friendly way?
I found a nice method DAGUtils.generateSimpleJSONPlan(dagPlan)
but the dagPlan can be obtained using a class (DAGImpl) implementing the
interface org.apache.tez.dag.app.DAG (the getJobPlan() method) , not
an instance of org.apache.tez.dag.api.DAG.
.
Thank You very much in advance.
Yours sincerely,
Camelia
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: How to display DAG?
Posted by camelia c <ca...@yahoo.com>.
I'd like to display it before launch in execution. Is there a method
available that traverses the dag obtained with the DAG API and displays
debug-like info, or should I write one myself?
Thank You for Your prompt answer.
Camelia
________________________________
From: Bikas Saha <bi...@hortonworks.com>
To: user@tez.incubator.apache.org; camelia c <ca...@yahoo.com>
Sent: Thursday, June 26, 2014 7:51 PM
Subject: RE: How to display DAG?
That method is probably used to generate history events for recovery.
When the dag runs, it generates a dot plot of the graph and writes it to its local dir. If you are running it locally then you can find it in the log dir of your application master container. If you are running on the cluster and have log aggregation turned on, then you can download it via the “yarn log” command. The file name would be like “dag_1403748209003_0006_1.dot”. Then you can view it using a dot plot viewing tool.
Bikas
From:camelia c [mailto:camelie_1985@yahoo.com]
Sent: Thursday, June 26, 2014 9:45 AM
To: user@tez.incubator.apache.org
Subject: How to display DAG?
Hello,
This is my second question for today.
Suppose that I managed to create the DAG successfully, how can I display the plan (with vertices and edges) in a user-friendly way?
I found a nice method DAGUtils.generateSimpleJSONPlan(dagPlan)
but the dagPlan can be obtained using a class (DAGImpl) implementing the interface org.apache.tez.dag.app.DAG (the getJobPlan() method) , not an instance of org.apache.tez.dag.api.DAG.
.
Thank You very much in advance.
Yours sincerely,
Camelia
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: How to display DAG?
Posted by Bikas Saha <bi...@hortonworks.com>.
That method is probably used to generate history events for recovery.
When the dag runs, it generates a dot plot of the graph and writes it to
its local dir. If you are running it locally then you can find it in the
log dir of your application master container. If you are running on the
cluster and have log aggregation turned on, then you can download it via
the “yarn log” command. The file name would be like
“dag_1403748209003_0006_1.dot”. Then you can view it using a dot plot
viewing tool.
Bikas
*From:* camelia c [mailto:camelie_1985@yahoo.com]
*Sent:* Thursday, June 26, 2014 9:45 AM
*To:* user@tez.incubator.apache.org
*Subject:* How to display DAG?
Hello,
This is my second question for today.
Suppose that I managed to create the DAG successfully, how can I display
the plan (with vertices and edges) in a user-friendly way?
I found a nice method DAGUtils.generateSimpleJSONPlan(dagPlan)
but the dagPlan can be obtained using a class (DAGImpl) implementing the
interface org.apache.tez.dag.app.DAG (the getJobPlan() method) , not
an instance of org.apache.tez.dag.api.DAG.
.
Thank You very much in advance.
Yours sincerely,
Camelia
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.