You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2009/06/25 12:03:07 UTC
[jira] Created: (PIG-864) Record graph of execution of Map-Reduce
jobs executed by a Pig script
Record graph of execution of Map-Reduce jobs executed by a Pig script
---------------------------------------------------------------------
Key: PIG-864
URL: https://issues.apache.org/jira/browse/PIG-864
Project: Pig
Issue Type: Improvement
Reporter: Arun C Murthy
It would be useful for offline analysis if Pig were to record the entire graph of Map-Reduce jobs executed by a singe Pig script.
For starters a simple 'parent jobid' for each MR job in the graph would be nice.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-864) Record graph of execution of Map-Reduce
jobs executed by a Pig script
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853039#action_12853039 ]
Dmitriy V. Ryaboy commented on PIG-864:
---------------------------------------
In that case... think you can knock out PIG-908 while you are in there? Most of the work, except the actual logging, has been done -- I added the relation names to Physical Operators in PIG-1156
> Record graph of execution of Map-Reduce jobs executed by a Pig script
> ---------------------------------------------------------------------
>
> Key: PIG-864
> URL: https://issues.apache.org/jira/browse/PIG-864
> Project: Pig
> Issue Type: Improvement
> Reporter: Arun C Murthy
> Assignee: Richard Ding
>
> It would be useful for offline analysis if Pig were to record the entire graph of Map-Reduce jobs executed by a singe Pig script.
> For starters a simple 'parent jobid' for each MR job in the graph would be nice.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-864) Record graph of execution of Map-Reduce
jobs executed by a Pig script
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852985#action_12852985 ]
Dmitriy V. Ryaboy commented on PIG-864:
---------------------------------------
Richard, is the idea to log something that carries this information, or to produce an actual DOT file with the graph?
> Record graph of execution of Map-Reduce jobs executed by a Pig script
> ---------------------------------------------------------------------
>
> Key: PIG-864
> URL: https://issues.apache.org/jira/browse/PIG-864
> Project: Pig
> Issue Type: Improvement
> Reporter: Arun C Murthy
> Assignee: Richard Ding
>
> It would be useful for offline analysis if Pig were to record the entire graph of Map-Reduce jobs executed by a singe Pig script.
> For starters a simple 'parent jobid' for each MR job in the graph would be nice.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-864) Record graph of execution of Map-Reduce
jobs executed by a Pig script
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding resolved PIG-864.
------------------------------
Fix Version/s: 0.8.0
Resolution: Fixed
PIG-1333 fixed this issue with
* A new entry "pig.parent.jobid" is added to every MR job that depends on other jobs. The value of this entry is a comma separated Job ids.
* The new API on PigRunner executes a Pig scripts and returns a PigStats object that contains a graph of execution of MR jobs.
> Record graph of execution of Map-Reduce jobs executed by a Pig script
> ---------------------------------------------------------------------
>
> Key: PIG-864
> URL: https://issues.apache.org/jira/browse/PIG-864
> Project: Pig
> Issue Type: Improvement
> Reporter: Arun C Murthy
> Assignee: Richard Ding
> Fix For: 0.8.0
>
>
> It would be useful for offline analysis if Pig were to record the entire graph of Map-Reduce jobs executed by a singe Pig script.
> For starters a simple 'parent jobid' for each MR job in the graph would be nice.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-864) Record graph of execution of Map-Reduce
jobs executed by a Pig script
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872744#action_12872744 ]
Richard Ding commented on PIG-864:
----------------------------------
The plan is to add a new entry--"pig.parent.jobid"--to each job which has parents, in addition to log the job DAG of a Pig script to the Pig log.
> Record graph of execution of Map-Reduce jobs executed by a Pig script
> ---------------------------------------------------------------------
>
> Key: PIG-864
> URL: https://issues.apache.org/jira/browse/PIG-864
> Project: Pig
> Issue Type: Improvement
> Reporter: Arun C Murthy
> Assignee: Richard Ding
>
> It would be useful for offline analysis if Pig were to record the entire graph of Map-Reduce jobs executed by a singe Pig script.
> For starters a simple 'parent jobid' for each MR job in the graph would be nice.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-864) Record graph of execution of Map-Reduce
jobs executed by a Pig script
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853031#action_12853031 ]
Richard Ding commented on PIG-864:
----------------------------------
I'm thinking about logging this information as part of implementation of PIG-1333.
> Record graph of execution of Map-Reduce jobs executed by a Pig script
> ---------------------------------------------------------------------
>
> Key: PIG-864
> URL: https://issues.apache.org/jira/browse/PIG-864
> Project: Pig
> Issue Type: Improvement
> Reporter: Arun C Murthy
> Assignee: Richard Ding
>
> It would be useful for offline analysis if Pig were to record the entire graph of Map-Reduce jobs executed by a singe Pig script.
> For starters a simple 'parent jobid' for each MR job in the graph would be nice.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-864) Record graph of execution of Map-Reduce
jobs executed by a Pig script
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853546#action_12853546 ]
Richard Ding commented on PIG-864:
----------------------------------
OK, I'll take PIG-908 and see if it can be knocked out :-)
> Record graph of execution of Map-Reduce jobs executed by a Pig script
> ---------------------------------------------------------------------
>
> Key: PIG-864
> URL: https://issues.apache.org/jira/browse/PIG-864
> Project: Pig
> Issue Type: Improvement
> Reporter: Arun C Murthy
> Assignee: Richard Ding
>
> It would be useful for offline analysis if Pig were to record the entire graph of Map-Reduce jobs executed by a singe Pig script.
> For starters a simple 'parent jobid' for each MR job in the graph would be nice.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-864) Record graph of execution of Map-Reduce
jobs executed by a Pig script
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding reassigned PIG-864:
--------------------------------
Assignee: Richard Ding
> Record graph of execution of Map-Reduce jobs executed by a Pig script
> ---------------------------------------------------------------------
>
> Key: PIG-864
> URL: https://issues.apache.org/jira/browse/PIG-864
> Project: Pig
> Issue Type: Improvement
> Reporter: Arun C Murthy
> Assignee: Richard Ding
>
> It would be useful for offline analysis if Pig were to record the entire graph of Map-Reduce jobs executed by a singe Pig script.
> For starters a simple 'parent jobid' for each MR job in the graph would be nice.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.