You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Shubham Chopra (JIRA)" <ji...@apache.org> on 2009/01/19 13:11:59 UTC

[jira] Created: (PIG-626) Statistics (records read by each mapper and reducer)

Statistics (records read by each mapper and reducer)
----------------------------------------------------

                 Key: PIG-626
                 URL: https://issues.apache.org/jira/browse/PIG-626
             Project: Pig
          Issue Type: New Feature
          Components: impl
    Affects Versions: types_branch
            Reporter: Shubham Chopra
            Priority: Minor


This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:

String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
        if(reducePlan == null) {
            System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
        } else {
            System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
        }
The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688676#action_12688676 ] 

Shubham Chopra commented on PIG-626:
------------------------------------

I tested the patch with the latest trunk and it worked fine.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 1.0.0
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shubham Chopra updated PIG-626:
-------------------------------

    Attachment: pigStats.patch

Patch compatible with the latest trunk attached.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708737#action_12708737 ] 

Hadoop QA commented on PIG-626:
-------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12407672/PIG-626.patch
  against trunk revision 774167.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 12 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/34/console

This message is automatically generated.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>            Priority: Minor
>             Fix For: 0.3.0
>
>         Attachments: PIG-626.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alan Gates updated PIG-626:
---------------------------

    Assignee: Shubham Chopra
      Status: Patch Available  (was: Open)

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12699798#action_12699798 ] 

Shubham Chopra commented on PIG-626:
------------------------------------

Tested the patch with the latest trunk.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alan Gates updated PIG-626:
---------------------------

       Resolution: Fixed
    Fix Version/s: 0.3.0
           Status: Resolved  (was: Patch Available)

Patch checked in.  Thanks Shubham for your patience on this one.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>            Priority: Minor
>             Fix For: 0.3.0
>
>         Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shubham Chopra updated PIG-626:
-------------------------------

    Attachment: pigStats.patch

PigStats patch.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shubham Chopra updated PIG-626:
-------------------------------

    Attachment: pigStats.patch

Patch attached.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alan Gates updated PIG-626:
---------------------------

    Status: Patch Available  (was: Reopened)

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>            Priority: Minor
>             Fix For: 0.3.0
>
>         Attachments: PIG-626.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alan Gates updated PIG-626:
---------------------------

    Attachment: TEST-org.apache.pig.test.TestBZip.txt

Results of TestBZIp

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705152#action_12705152 ] 

Alan Gates commented on PIG-626:
--------------------------------

Shubham,

I apologize for being so slow to get to this.  I see several issues in the patch.

1) The launcher code has changed significantly.  I tried to figure out how to paste your code in, but I was afraid I was doing it wrong so I backed off.  You'll need to integrate your patch with the changes.

2) All error messages now require an error code (see http://wiki.apache.org/pig/PigErrorHandlingFunctionalSpecification).

3) In a couple places when the code can't find the statistics information it says:

{code}
 log.info("Jobs not found in the JobClient. Please try using a different Hadoop mode"); 
{code}

It seems like this ought to be a warning instead of an info.  Also, what does it mean to try a different hadoop mode?  It's not clear from this message what action the user should take.

I promise if you make another patch for this I'll look at it quickly so it doesn't go out of date on you again.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665512#action_12665512 ] 

Alan Gates commented on PIG-626:
--------------------------------

Shubham, I have several comments and questions about this patch:

# What exactly is the goal here, to make statistics available to users via the PigServer interface, or to add certain specific statistics to map and reduce?  
# If I understand correctly you are just returning pig specific counters in PigServer.getStats().  Why not return all counters (if they are available)?  Perhaps users would want to read the hadoop counters too.
# There are some println's stuck in here that print out the stats at the end of the job.  This should be written to the logs instead.
# I don't see where the statistics are actually being incremented during the map reduce job.  I see where they're collected after the job.


> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shubham Chopra updated PIG-626:
-------------------------------

    Attachment: pigStats.patch


1. The initial goal is to just get the pig query result sizes. I try to get the no. of records written by the last job which gives me the required metric. In addition to this I am making the stats for all the jobs available. I am not adding any specific stat to map/reduce.
2. I am only using the hadoop counters here. I am not registering any pig-specific counters. Olga had suggested that I make the counters available in the form of a map with the keys starting with PIG_STATS to reference the required counter. 
Hadoop has a bunch of counters that it uses to display stats in its map-red webUI. These counters give the no. of records and bytes read/written by map/reduce of each job. I am making all the hadoop stats, at the granularity of a record, available.
3. I have rectified this in the new patch.
4. These counters are incremented within hadoop.


> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Rekha (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717946#action_12717946 ] 

Rekha commented on PIG-626:
---------------------------

Shubham/Alan,

This is nice.I need to trap the below data per script and save it into database as record of the run.

PIG_STATS_BYTES_WRITTEN
PIG_STATS_RECORDS_WRITTEN
PIG_STATS_MAP_INPUT_RECORDS,
PIG_STATS_MAP_OUTPUT_RECORDS, 
PIG_STATS_REDUCE_INPUT_RECORDS, 
PIG_STATS_REDUCE_OUTPUT_RECORDS, 
PIG_STATS_REDUCE_PLAN, 
PIG_STATS_COMBINE_PLAN

In pig 2.2 I see similar info in the console logs.Can the pulling be done from there?

In anycase , I do not want stats run to be an offline process as seen in the Test files in the package.
Ideally would like to call/execute the stats module right from pig scripts itself as -
stats.record(); //first line
stats.flush('out<jobid>.txt');//last line

The push to db can be a cron process.

Any suggestions?

Thanks & Regards,
/Rekha.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>            Priority: Minor
>             Fix For: 0.3.0
>
>         Attachments: PIG-626.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Reopened: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alan Gates reopened PIG-626:
----------------------------


I should have checked my other window before I marked the bug as fixed.  The commit failed, I can't seem to contact Apache's SVN at the moment.  I'll commit the patch once I can.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>            Priority: Minor
>             Fix For: 0.3.0
>
>         Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668678#action_12668678 ] 

Alan Gates commented on PIG-626:
--------------------------------

I ran the unit tests on the latest patch and saw failures in TestBZip.  It's getting a null pointer exception when it tries to reference PigStats at some point.  I'll attach the results to the bug.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shubham Chopra updated PIG-626:
-------------------------------

    Attachment: pigStats.patch

Updates pigStats patch. I now make use of the ExecJob interface.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 1.0.0
>            Reporter: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alan Gates updated PIG-626:
---------------------------

    Attachment: PIG-626.patch

A version of the patch that deals with the findbugs and javac warnings.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>            Priority: Minor
>             Fix For: 0.3.0
>
>         Attachments: PIG-626.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alan Gates updated PIG-626:
---------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Patch committed, for real this time.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>            Priority: Minor
>             Fix For: 0.3.0
>
>         Attachments: PIG-626.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-626) Statistics (records read by each mapper and reducer)

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707432#action_12707432 ] 

Hadoop QA commented on PIG-626:
-------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12407613/pigStats.patch
  against trunk revision 772750.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 12 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 226 javac compiler warnings (more than the trunk's current 225 warnings).

    -1 findbugs.  The patch appears to introduce 2 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/33/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/33/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/33/console

This message is automatically generated.

> Statistics (records read by each mapper and reducer)
> ----------------------------------------------------
>
>                 Key: PIG-626
>                 URL: https://issues.apache.org/jira/browse/PIG-626
>             Project: Pig
>          Issue Type: New Feature
>          Components: impl
>    Affects Versions: 0.2.0
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>            Priority: Minor
>         Attachments: pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, pigStats.patch, TEST-org.apache.pig.test.TestBZip.txt
>
>
> This uses the counters framework that hadoop has. Initially, I am just interested in finding out the number of records read by each mapper/reducer particularly for the last job in any script. A sample code to access the statistics for the last job:
> String reducePlan = stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_PLAN");
>         if(reducePlan == null) {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_MAP_OUTPUT_RECORDS"));
>         } else {
>             System.out.println("Records written : " + stats.getPigStats().get(stats.getLastJobID()).get("PIG_STATS_REDUCE_OUTPUT_RECORDS"));
>         }
> The patch contains 7 test cases. These include tests PigStorage and BinStorage along with one for multiple MR jobs case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.