You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Shubham Chopra (JIRA)" <ji...@apache.org> on 2009/01/19 13:11:59 UTC
[jira] Created: (PIG-626) Statistics (records read by each mapper
and reducer)
Statistics (records read by each mapper and reducer)
----------------------------------------------------
Key: PIG-626
URL: https://issues.apache.org/jira/browse/PIG-626
Project: Pig
Issue Type: New Feature
Components: impl
Affects Versions: types_branch
Reporter: Shubham Chopra
Priority: Minor
This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
if(reducePlan == null) {
System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
} else {
System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
}
The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688676#action_12688676 ]
Shubham Chopra commented on PIG-626:
------------------------------------
I tested the patch with the latest trunk and it worked fine.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shubham Chopra updated PIG-626:
-------------------------------
Attachment: pigStats.patch
Patch compatible with the latest trunk attached.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708737#action_12708737 ]
Hadoop QA commented on PIG-626:
-------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12407672/PIG-626.patch
against trunk revision 774167.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/34/console
This message is automatically generated.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Assignee: Shubham Chopra
> Priority: Minor
> Fix For: 0.3.0
>
> Attachments: PIG-626.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-626:
---------------------------
Assignee: Shubham Chopra
Status: Patch Available (was: Open)
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Assignee: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699798#action_12699798 ]
Shubham Chopra commented on PIG-626:
------------------------------------
Tested the patch with the latest trunk.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-626:
---------------------------
Resolution: Fixed
Fix Version/s: 0.3.0
Status: Resolved (was: Patch Available)
Patch checked in. Thanks Shubham for your patience on this one.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Assignee: Shubham Chopra
> Priority: Minor
> Fix For: 0.3.0
>
> Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shubham Chopra updated PIG-626:
-------------------------------
Attachment: pigStats.patch
PigStats patch.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: types_branch
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shubham Chopra updated PIG-626:
-------------------------------
Attachment: pigStats.patch
Patch attached.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: types_branch
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-626:
---------------------------
Status: Patch Available (was: Reopened)
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Assignee: Shubham Chopra
> Priority: Minor
> Fix For: 0.3.0
>
> Attachments: PIG-626.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-626:
---------------------------
Attachment: TEST-org.apache.pig.test.TestBZip.txt
Results of TestBZIp
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: types_branch
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705152#action_12705152 ]
Alan Gates commented on PIG-626:
--------------------------------
Shubham,
I apologize for being so slow to get to this. I see several issues in the patch.
1) The launcher code has changed significantly. I tried to figure out how to paste your code in, but I was afraid I was doing it wrong so I backed off. You'll need to integrate your patch with the changes.
2) All error messages now require an error code (see http://wiki.apache.org/pig/PigErrorHandlingFunctionalSpecification).
3) In a couple places when the code can't find the statistics information it says:
{code}
log.info("Jobs not found in the JobClient. Please try using a different Hadoop mode");
{code}
It seems like this ought to be a warning instead of an info. Also, what does it mean to try a different hadoop mode? It's not clear from this message what action the user should take.
I promise if you make another patch for this I'll look at it quickly so it doesn't go out of date on you again.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665512#action_12665512 ]
Alan Gates commented on PIG-626:
--------------------------------
Shubham, I have several comments and questions about this patch:
# What exactly is the goal here, to make statistics available to users via the PigServer interface, or to add certain specific statistics to map and reduce?
# If I understand correctly you are just returning pig specific counters in PigServer.getStats(). Why not return all counters (if they are available)? Perhaps users would want to read the hadoop counters too.
# There are some println's stuck in here that print out the stats at the end of the job. This should be written to the logs instead.
# I don't see where the statistics are actually being incremented during the map reduce job. I see where they're collected after the job.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: types_branch
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shubham Chopra updated PIG-626:
-------------------------------
Attachment: pigStats.patch
1. The initial goal is to just get the pig query result sizes. I try to get the no. of records written by the last job which gives me the required metric. In addition to this I am making the stats for all the jobs available. I am not adding any specific stat to map/reduce.
2. I am only using the hadoop counters here. I am not registering any pig-specific counters. Olga had suggested that I make the counters available in the form of a map with the keys starting with PIG_STATS to reference the required counter.
Hadoop has a bunch of counters that it uses to display stats in its map-red webUI. These counters give the no. of records and bytes read/written by map/reduce of each job. I am making all the hadoop stats, at the granularity of a record, available.
3. I have rectified this in the new patch.
4. These counters are incremented within hadoop.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: types_branch
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Rekha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717946#action_12717946 ]
Rekha commented on PIG-626:
---------------------------
Shubham/Alan,
This is nice.I need to trap the below data per script and save it into database as record of the run.
PIG_STATS_BYTES_WRITTEN
PIG_STATS_RECORDS_WRITTEN
PIG_STATS_MAP_INPUT_RECORDS,
PIG_STATS_MAP_OUTPUT_RECORDS,
PIG_STATS_REDUCE_INPUT_RECORDS,
PIG_STATS_REDUCE_OUTPUT_RECORDS,
PIG_STATS_REDUCE_PLAN,
PIG_STATS_COMBINE_PLAN
In pig 2.2 I see similar info in the console logs.Can the pulling be done from there?
In anycase , I do not want stats run to be an offline process as seen in the Test files in the package.
Ideally would like to call/execute the stats module right from pig scripts itself as -
stats.record(); //first line
stats.flush('out<jobid>.txt');//last line
The push to db can be a cron process.
Any suggestions?
Thanks & Regards,
/Rekha.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Assignee: Shubham Chopra
> Priority: Minor
> Fix For: 0.3.0
>
> Attachments: PIG-626.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates reopened PIG-626:
----------------------------
I should have checked my other window before I marked the bug as fixed. The commit failed, I can't seem to contact Apache's SVN at the moment. I'll commit the patch once I can.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Assignee: Shubham Chopra
> Priority: Minor
> Fix For: 0.3.0
>
> Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668678#action_12668678 ]
Alan Gates commented on PIG-626:
--------------------------------
I ran the unit tests on the latest patch and saw failures in TestBZip. It's getting a null pointer exception when it tries to reference PigStats at some point. I'll attach the results to the bug.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: types_branch
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shubham Chopra updated PIG-626:
-------------------------------
Attachment: pigStats.patch
Updates pigStats patch. I now make use of the ExecJob interface.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-626:
---------------------------
Attachment: PIG-626.patch
A version of the patch that deals with the findbugs and javac warnings.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Assignee: Shubham Chopra
> Priority: Minor
> Fix For: 0.3.0
>
> Attachments: PIG-626.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-626:
---------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch committed, for real this time.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Assignee: Shubham Chopra
> Priority: Minor
> Fix For: 0.3.0
>
> Attachments: PIG-626.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-626) Statistics (records read by each mapper
and reducer)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707432#action_12707432 ]
Hadoop QA commented on PIG-626:
-------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12407613/pigStats.patch
against trunk revision 772750.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 226 javac compiler warnings (more than the trunk's current 225 warnings).
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/33/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/33/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/33/console
This message is automatically generated.
> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
> Key: PIG-626
> URL: https://issues.apache.org/jira/browse/PIG-626
> Project: Pig
> Issue Type: New Feature
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Shubham Chopra
> Assignee: Shubham Chopra
> Priority: Minor
> Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
> if(reducePlan == null) {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
> } else {
> System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
> }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.