You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org> on 2009/08/04 21:56:15 UTC
[jira] Created: (PIG-908) Need a way to correlate MR jobs with Pig
statements
Need a way to correlate MR jobs with Pig statements
---------------------------------------------------
Key: PIG-908
URL: https://issues.apache.org/jira/browse/PIG-908
Project: Pig
Issue Type: Wish
Reporter: Dmitriy V. Ryaboy
Complex Pig Scripts often generate many Map-Reduce jobs, especially with the recent introduction of multi-store capabilities.
For example, the first script in the Pig tutorial produces 5 MR jobs.
There is currently very little support for debugging resulting jobs; if one of the MR jobs fails, it is hard to figure out which part of the script it was responsible for. Explain plans help, but even with the explain plan, a fair amount of effort (and sometimes, experimentation) is required to correlate the failing MR job with the corresponding PigLatin statements.
This ticket is created to discuss approaches to alleviating this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-908) Need a way to correlate MR jobs with
Pig statements
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881828#action_12881828 ]
Richard Ding commented on PIG-908:
----------------------------------
It's hard to correlate MR jobs with line numbers in Pig script in the current implementation. So we decided that the next best thing is to correlate MR jobs with aliases defined in Pig script.
PIG-1333 added "pig.alias" to the MR jobs so it can be viewed in Job xml. The value of "pig.alias" is a comma-separated list of aliases since a MR job can be composed of several Pig statements.
> Need a way to correlate MR jobs with Pig statements
> ---------------------------------------------------
>
> Key: PIG-908
> URL: https://issues.apache.org/jira/browse/PIG-908
> Project: Pig
> Issue Type: Wish
> Reporter: Dmitriy V. Ryaboy
> Assignee: Richard Ding
> Fix For: 0.8.0
>
>
> Complex Pig Scripts often generate many Map-Reduce jobs, especially with the recent introduction of multi-store capabilities.
> For example, the first script in the Pig tutorial produces 5 MR jobs.
> There is currently very little support for debugging resulting jobs; if one of the MR jobs fails, it is hard to figure out which part of the script it was responsible for. Explain plans help, but even with the explain plan, a fair amount of effort (and sometimes, experimentation) is required to correlate the failing MR job with the corresponding PigLatin statements.
> This ticket is created to discuss approaches to alleviating this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-908) Need a way to correlate MR jobs with
Pig statements
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739125#action_12739125 ]
Dmitriy V. Ryaboy commented on PIG-908:
---------------------------------------
An idea for something might work (haven't evaluated the complexity of implementing this)
When LogicalOperators are created, a bit of metadata is attached to them, listing the line number that they come from. Multiple LOs may be created from a single line, and multiple lines may be associated with a single operator.
This metadata is passed down to Physical Operators.
When an MR job is created, a log message is written listing the line numbers that are associated with the POs in this map-reduce job, and the job name.
Thoughts?
> Need a way to correlate MR jobs with Pig statements
> ---------------------------------------------------
>
> Key: PIG-908
> URL: https://issues.apache.org/jira/browse/PIG-908
> Project: Pig
> Issue Type: Wish
> Reporter: Dmitriy V. Ryaboy
>
> Complex Pig Scripts often generate many Map-Reduce jobs, especially with the recent introduction of multi-store capabilities.
> For example, the first script in the Pig tutorial produces 5 MR jobs.
> There is currently very little support for debugging resulting jobs; if one of the MR jobs fails, it is hard to figure out which part of the script it was responsible for. Explain plans help, but even with the explain plan, a fair amount of effort (and sometimes, experimentation) is required to correlate the failing MR job with the corresponding PigLatin statements.
> This ticket is created to discuss approaches to alleviating this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-908) Need a way to correlate MR jobs with Pig
statements
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-908:
-------------------------------
With Pig 0.8.0 we print a summary of the execution that contains (among other things) how aliases mapped to jobs. Example:
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
job_201004271216_12712 1 1 3 3 3 12 12 12 B,C GROUP_BY,COMBINER
job_201004271216_12713 1 1 3 3 3 12 12 12 D SAMPLER
job_201004271216_12714 1 1 3 3 3 12 12 12 D ORDER_BY,COMBINER hdfs://wilbur20.labs.corp.sp1.yahoo.com:9020/tmp/temp743703298/tmp-2019944040,
> Need a way to correlate MR jobs with Pig statements
> ---------------------------------------------------
>
> Key: PIG-908
> URL: https://issues.apache.org/jira/browse/PIG-908
> Project: Pig
> Issue Type: Wish
> Reporter: Dmitriy V. Ryaboy
> Assignee: Richard Ding
> Fix For: 0.8.0
>
>
> Complex Pig Scripts often generate many Map-Reduce jobs, especially with the recent introduction of multi-store capabilities.
> For example, the first script in the Pig tutorial produces 5 MR jobs.
> There is currently very little support for debugging resulting jobs; if one of the MR jobs fails, it is hard to figure out which part of the script it was responsible for. Explain plans help, but even with the explain plan, a fair amount of effort (and sometimes, experimentation) is required to correlate the failing MR job with the corresponding PigLatin statements.
> This ticket is created to discuss approaches to alleviating this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-908) Need a way to correlate MR jobs with Pig
statements
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding reassigned PIG-908:
--------------------------------
Assignee: Richard Ding
> Need a way to correlate MR jobs with Pig statements
> ---------------------------------------------------
>
> Key: PIG-908
> URL: https://issues.apache.org/jira/browse/PIG-908
> Project: Pig
> Issue Type: Wish
> Reporter: Dmitriy V. Ryaboy
> Assignee: Richard Ding
>
> Complex Pig Scripts often generate many Map-Reduce jobs, especially with the recent introduction of multi-store capabilities.
> For example, the first script in the Pig tutorial produces 5 MR jobs.
> There is currently very little support for debugging resulting jobs; if one of the MR jobs fails, it is hard to figure out which part of the script it was responsible for. Explain plans help, but even with the explain plan, a fair amount of effort (and sometimes, experimentation) is required to correlate the failing MR job with the corresponding PigLatin statements.
> This ticket is created to discuss approaches to alleviating this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-908) Need a way to correlate MR jobs with Pig
statements
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding resolved PIG-908.
------------------------------
Resolution: Fixed
With PIG-1333, I mark this Jira as fixed. Please reopen the jira if you think PIG-1333 doesn't provide what you need.
> Need a way to correlate MR jobs with Pig statements
> ---------------------------------------------------
>
> Key: PIG-908
> URL: https://issues.apache.org/jira/browse/PIG-908
> Project: Pig
> Issue Type: Wish
> Reporter: Dmitriy V. Ryaboy
> Assignee: Richard Ding
> Fix For: 0.8.0
>
>
> Complex Pig Scripts often generate many Map-Reduce jobs, especially with the recent introduction of multi-store capabilities.
> For example, the first script in the Pig tutorial produces 5 MR jobs.
> There is currently very little support for debugging resulting jobs; if one of the MR jobs fails, it is hard to figure out which part of the script it was responsible for. Explain plans help, but even with the explain plan, a fair amount of effort (and sometimes, experimentation) is required to correlate the failing MR job with the corresponding PigLatin statements.
> This ticket is created to discuss approaches to alleviating this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-908) Need a way to correlate MR jobs with
Pig statements
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739147#action_12739147 ]
Santhosh Srinivasan commented on PIG-908:
-----------------------------------------
+1
This approach has been discussed but not documented.
> Need a way to correlate MR jobs with Pig statements
> ---------------------------------------------------
>
> Key: PIG-908
> URL: https://issues.apache.org/jira/browse/PIG-908
> Project: Pig
> Issue Type: Wish
> Reporter: Dmitriy V. Ryaboy
>
> Complex Pig Scripts often generate many Map-Reduce jobs, especially with the recent introduction of multi-store capabilities.
> For example, the first script in the Pig tutorial produces 5 MR jobs.
> There is currently very little support for debugging resulting jobs; if one of the MR jobs fails, it is hard to figure out which part of the script it was responsible for. Explain plans help, but even with the explain plan, a fair amount of effort (and sometimes, experimentation) is required to correlate the failing MR job with the corresponding PigLatin statements.
> This ticket is created to discuss approaches to alleviating this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-908) Need a way to correlate MR jobs with Pig
statements
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-908:
-------------------------------
Fix Version/s: 0.8.0
> Need a way to correlate MR jobs with Pig statements
> ---------------------------------------------------
>
> Key: PIG-908
> URL: https://issues.apache.org/jira/browse/PIG-908
> Project: Pig
> Issue Type: Wish
> Reporter: Dmitriy V. Ryaboy
> Assignee: Richard Ding
> Fix For: 0.8.0
>
>
> Complex Pig Scripts often generate many Map-Reduce jobs, especially with the recent introduction of multi-store capabilities.
> For example, the first script in the Pig tutorial produces 5 MR jobs.
> There is currently very little support for debugging resulting jobs; if one of the MR jobs fails, it is hard to figure out which part of the script it was responsible for. Explain plans help, but even with the explain plan, a fair amount of effort (and sometimes, experimentation) is required to correlate the failing MR job with the corresponding PigLatin statements.
> This ticket is created to discuss approaches to alleviating this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.