You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Ashish Vaidya (JIRA)" <ji...@apache.org> on 2012/07/11 01:51:33 UTC
[jira] [Created] (OOZIE-906) Show runtime job DAG visually in Oozie
console/dashboard
Ashish Vaidya created OOZIE-906:
-----------------------------------
Summary: Show runtime job DAG visually in Oozie console/dashboard
Key: OOZIE-906
URL: https://issues.apache.org/jira/browse/OOZIE-906
Project: Oozie
Issue Type: New Feature
Reporter: Ashish Vaidya
The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
- Job Info
- Job Definition
- Job Configuration
- Job Log
It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (OOZIE-906) Show runtime job DAG visually in
Oozie console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Vaidya reassigned OOZIE-906:
-----------------------------------
Assignee: Ashish Vaidya
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-906) Show runtime job DAG visually in Oozie
console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Vaidya updated OOZIE-906:
--------------------------------
Attachment: OOZIE-906.v2.patch
graphWF.xml
TestGraphGenerator.java
GraphGenerator.java
Newer patch with tests
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: GraphGenerator.java, GraphGenerator.java, OOZIE-906.patch, OOZIE-906.v2.patch, Sample runtime DAG.png, TestGraphGenerator.java, graphFRShape.png, graphWF.xml
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-906) Show runtime job DAG visually in Oozie
console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Vaidya updated OOZIE-906:
--------------------------------
Attachment: OOZIE-906.patch
First cut. Comments, critiques, and suggestions for improving this are most welcome.
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: OOZIE-906.patch, graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-906) Show runtime job DAG visually in
Oozie console/dashboard
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429607#comment-13429607 ]
Robert Kanter commented on OOZIE-906:
-------------------------------------
Good Job, that looks really great.
Two things:
1) Perhaps we should change some of the colors? The bright red is kinda hard of the eyes and the dark grey on black for the end node makes it difficult to read.
2) How "expensive" is it to generate the graph image? It looks like you generate a new graph each time a request is made, so I'd be concerned that someone spamming the refresh button could cause a problem on a large workflow. Is there some way to save the graphs once they are generated and then just update them when things change?
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: GraphGenerator.java, OOZIE-906.patch, Sample runtime DAG.png, graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-906) Show runtime job DAG visually in
Oozie console/dashboard
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429934#comment-13429934 ]
Alejandro Abdelnur commented on OOZIE-906:
------------------------------------------
Ashish, nice. Note that OOZIE-243 has been committed, you know don't need to get the control nodes, they show up as action nodes of special type.
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: GraphGenerator.java, OOZIE-906.patch, Sample runtime DAG.png, graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-906) Show runtime job DAG visually in Oozie
console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Vaidya updated OOZIE-906:
--------------------------------
Attachment: Sample runtime DAG.png
A sample runtime DAG PNG
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: OOZIE-906.patch, Sample runtime DAG.png, graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-906) Show runtime job DAG visually in
Oozie console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429924#comment-13429924 ]
Ashish Vaidya commented on OOZIE-906:
-------------------------------------
Oh and the rendering of the graph isn't completely foolproof. If given a sufficiently complex workflow, my simplistic node placement algorithm can't guarantee that overlaps won't happen!
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: GraphGenerator.java, OOZIE-906.patch, Sample runtime DAG.png, graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-906) Show runtime job DAG visually in Oozie
console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Vaidya updated OOZIE-906:
--------------------------------
Attachment: GraphGenerator.java
Forgot the main code file :-P
The code here needs to be made workflow XML verstion aware to handle various XML versions that the job definitions may have.
As this graph is rendered after the Oozie has accepted the XML, no XML validation is done while generating it.
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: GraphGenerator.java, OOZIE-906.patch, Sample runtime DAG.png, graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-906) Show runtime job DAG visually in
Oozie console/dashboard
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422412#comment-13422412 ]
Alejandro Abdelnur commented on OOZIE-906:
------------------------------------------
Nice
OOZIE-243 is adding the control nodes (START/END/KILL/FORK/JOIN) to the DB, this means it will be easier to tag those node as 'executed'.
If somehow we could order the nodes rendering by start time (from left to right) it would be great.
thx
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-906) Show runtime job DAG visually in Oozie
console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Vaidya updated OOZIE-906:
--------------------------------
Attachment: graphFRShape.png
This is the graph I obtained using JUNG 2.0.1 library. It's real workflow that's being used in one of our products. To prevent giving away details of the product I've used action type to label the vertices of the graph. In the final code action names will be used.
The graph will paint the vertices according to the status of the actions at the time of rendering. Thus, an OK will be green, PREP or RUNNING will be yellow, KILLED, ERROR or FAILED will be RED. The nodes for which no info is yet available will be painted light grey. For now I've just painted the graph to highlight the various actions, but going forward it'll be per above coloring scheme.
I've used FRLayout algorithm to chart the graph. This algo seems to do a good job save for the fact that the orientation of the graph is not deterministic. Thus, the same graph when rendered again shows up differently. If anyone out there has any ideas to get this right please ping me on this JIRA.
Other comments and suggestions are most welcome.
Thanks,
Ashish
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-906) Show runtime job DAG visually in Oozie
console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Vaidya updated OOZIE-906:
--------------------------------
Attachment: OOZIE-906.v7.patch
Final patch now that the review had been 'all clear.'
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: graphFRShape.png, GraphGenerator.java, GraphGenerator.java, graphWF.xml, OOZIE-906.patch, OOZIE-906.v2.patch, OOZIE-906.v7.patch, Sample runtime DAG.png, TestGraphGenerator.java
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-906) Show runtime job DAG visually in
Oozie console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433234#comment-13433234 ]
Ashish Vaidya commented on OOZIE-906:
-------------------------------------
Review requested at <https://reviews.apache.org/r/6525/>.
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: graphFRShape.png, GraphGenerator.java, GraphGenerator.java, graphWF.xml, OOZIE-906.patch, OOZIE-906.v2.patch, Sample runtime DAG.png, TestGraphGenerator.java
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-906) Show runtime job DAG visually in
Oozie console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422422#comment-13422422 ]
Ashish Vaidya commented on OOZIE-906:
-------------------------------------
Ordering is something that I've no control over. Hence the request for suggesting any better algorithm. I can surely include start and end times, and any other metadata for each node.
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-906) Show runtime job DAG visually in
Oozie console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429571#comment-13429571 ]
Ashish Vaidya commented on OOZIE-906:
-------------------------------------
I take back my previous comment. I figured out a way to order the nodes per their order of occurance in the worflow XML. I'm attaching the code patch of the work so far.
- The REST API to access the graph is v1/job/{jobid}?show=graph. This is not, an perhaps won't be, available in v0 of the REST API.
- The graph is rendered from top to bottom, and delivered to the client in PNG format.
- The nodes that haven't yet started execution are painted gray. The nodes that are prepping or running are painted yellow. The nodes that have completed their execution successfully are painted green, and the ones that have failed/errored out are painted red.
- The error target (typically a kill node) is shown with dashed arrow. If we reach this kill node, it's painted red.
- I found that fork node doesn't carry any execution info. Thus, it remains painted gray even if it has been executed.
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-906) Show runtime job DAG visually in
Oozie console/dashboard
Posted by "Ashish Vaidya (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429922#comment-13429922 ]
Ashish Vaidya commented on OOZIE-906:
-------------------------------------
Thanks for the inputs, Robert. We can always decide on an eye-pleasing color scheme. I just picked the colors that are available readily in Java Color class.
As for the expense of generating it, it depends on how complex the workflow is. The very first picture that I posted to this JIRA is from a workflow XML that's over 2500 lines, and certainly takes a handful of seconds to generate and is quite a bit in size. While, some others are single node workflows which are very easily generated.
I'm not aware of any webserver level response caching mechanism in Oozie. If one exists then we can certainly do some timestamp comparison to avoid regenerating the graph if nothing has changed.
One thing that I found strange is the REST API assumes the output will be JSON for most parts. Writing PNG generation API using a class derived from JsonRestServlet seems a bit unnatural... but I digress.
> Show runtime job DAG visually in Oozie console/dashboard
> --------------------------------------------------------
>
> Key: OOZIE-906
> URL: https://issues.apache.org/jira/browse/OOZIE-906
> Project: Oozie
> Issue Type: New Feature
> Reporter: Ashish Vaidya
> Assignee: Ashish Vaidya
> Labels: graphical, job_graph, runtime_job_graph
> Attachments: GraphGenerator.java, OOZIE-906.patch, Sample runtime DAG.png, graphFRShape.png
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> The browser based Oozie console (the one currently written in ExtJS) shows the following tags when one clicks on a job to see its details:
> - Job Info
> - Job Definition
> - Job Configuration
> - Job Log
> It is desired to add a visual representation of the job DAG. An additional tag (likely called Job Graph) will be provided in the above console that will, when clicked, show the job workflow diagrammatically.
> The nodes in the graph/diagram will be color coded to represent their current state (Red for error/kill/failure, Green for OK and Yellow for the rest; or some combination thereof).
> The nodes will have names as provided in the job definition. The nodes may also be of different shapes (per action/control node) for better visibility. They may also carry some basic info like start time and duration.
> As regards implementation, it will be a server side with an HTTP API URL as invoker. The graphing may be done using GraphViz APIs, provided we find those to be Apache License compliant.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira