You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ankit Modi (JIRA)" <ji...@apache.org> on 2009/11/13 23:59:39 UTC
[jira] Created: (MAPREDUCE-1214) Add support for counters in Hadoop
Local Mode
Add support for counters in Hadoop Local Mode
---------------------------------------------
Key: MAPREDUCE-1214
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Ankit Modi
Currently there is no support for counters ( Records and Bytes written ) in Hadoop Local Mode.
Pig requires to provide counters to user when running in Hadoop Local Mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1214) Add support for counters in
Hadoop Local Mode
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791293#action_12791293 ]
Ankit Modi commented on MAPREDUCE-1214:
---------------------------------------
Hey Jeff,
Looks like this patch is breaking couple of things.
I'll look into whats breaking and see if this patch is useful.
> Add support for counters in Hadoop Local Mode
> ---------------------------------------------
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ankit Modi
> Attachments: MAPREDUCE-1214.patch
>
>
> Currently there is no support for counters ( Records and Bytes written ) in Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1214) Add support for counters in Hadoop
Local Mode
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankit Modi updated MAPREDUCE-1214:
----------------------------------
Attachment: MAPREDUCE-1214.2.patch
Hi Jeff,
Here is another patch based on suggestions provided in your patch. It does not change any api's.
The only issue pending in this is number of bytes written. The value that gets reported here is very high.
> Add support for counters in Hadoop Local Mode
> ---------------------------------------------
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ankit Modi
> Attachments: MAPREDUCE-1214.2.patch, MAPREDUCE-1214.patch
>
>
> Currently there is no support for counters ( Records and Bytes written ) in Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1214) Add support for counters in
Hadoop Local Mode
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790166#action_12790166 ]
Jeff Zhang commented on MAPREDUCE-1214:
---------------------------------------
Hi Ankit,
I checked the code in pig, and find that you use the JobClient the HExecutionEngine, but actually the JobClient in org.apache.hadoop.mapred.jobcontrol.Job submit the job. And each JobClient will create a LocalJobRunner, so that means you use the wrong JobClient/LocalJobRunner to get RunningJob. This won't affect the mapreduce mode, because you only have one master node as the JobTracker.
Forgive me to attach the Pig patch here.
Ankit, do you have the JIRA for this issue in Pig, I can move the patch there.
> Add support for counters in Hadoop Local Mode
> ---------------------------------------------
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ankit Modi
> Attachments: MAPREDUCE-1214.patch
>
>
> Currently there is no support for counters ( Records and Bytes written ) in Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1214) Add support for counters in Hadoop
Local Mode
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated MAPREDUCE-1214:
----------------------------------
Attachment: MAPREDUCE-1214.patch
> Add support for counters in Hadoop Local Mode
> ---------------------------------------------
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ankit Modi
> Attachments: MAPREDUCE-1214.patch
>
>
> Currently there is no support for counters ( Records and Bytes written ) in Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1214) Add support for counters in
Hadoop Local Mode
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791706#action_12791706 ]
Jeff Zhang commented on MAPREDUCE-1214:
---------------------------------------
Hi Ankit, what's problem with this patch ? Actually it works fine on my machine.
> Add support for counters in Hadoop Local Mode
> ---------------------------------------------
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ankit Modi
> Attachments: MAPREDUCE-1214.patch
>
>
> Currently there is no support for counters ( Records and Bytes written ) in Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1214) Add support for counters in
Hadoop Local Mode
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789973#action_12789973 ]
Jeff Zhang commented on MAPREDUCE-1214:
---------------------------------------
Ankit,
I have tried to use counters in hadoop local mode, and it works. Also custom counters is supported in hadoop local mode. Hope I did not misunderstand your meaning.
Here's my code snippet and output
{code}
public class MapReduceExample extends Configured implements Tool {
static class MyMapper extends Mapper<LongWritable, Text, LongWritable, Text> {
public MyMapper(){
}
protected void map(
LongWritable key,
Text value,
org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, LongWritable, Text>.Context context)
throws java.io.IOException, InterruptedException {
context.getCounter("mygroup", "jeff").increment(1);
context.write(key, value);
};
}
@Override
public int run(String[] args) throws Exception {
Job job = new Job();
job.setMapperClass(MyMapper.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
return 0;
}
public static void main(String[] args) throws Exception {
FileUtils.deleteDirectory(new File("data/output"));
args = new String[] { "data/input", "data/output" };
ToolRunner.run(new MapReduceExample(), args);
}
}
{code}
{code}
2009-12-14 11:38:41,843 INFO mapred.JobClient (JobClient.java:monitorAndPrintJob(1288)) - map 100% reduce 100%
2009-12-14 11:38:41,843 INFO mapred.JobClient (JobClient.java:monitorAndPrintJob(1343)) - Job complete: job_local_0001
2009-12-14 11:38:41,843 INFO mapred.JobClient (Counters.java:log(514)) - Counters: 13
2009-12-14 11:38:41,843 INFO mapred.JobClient (Counters.java:log(516)) - FileSystemCounters
2009-12-14 11:38:41,843 INFO mapred.JobClient (Counters.java:log(518)) - FILE_BYTES_READ=26096
2009-12-14 11:38:41,843 INFO mapred.JobClient (Counters.java:log(518)) - FILE_BYTES_WRITTEN=52792
2009-12-14 11:38:41,843 INFO mapred.JobClient (Counters.java:log(516)) - mygroup
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - jeff=2
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(516)) - Map-Reduce Framework
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - Reduce input groups=0
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - Combine output records=0
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - Map input records=2
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - Reduce shuffle bytes=0
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - Reduce output records=0
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - Spilled Records=4
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - Map output bytes=28
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - Combine input records=0
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - Map output records=2
2009-12-14 11:38:41,858 INFO mapred.JobClient (Counters.java:log(518)) - Reduce input records=2
{code}
> Add support for counters in Hadoop Local Mode
> ---------------------------------------------
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ankit Modi
>
> Currently there is no support for counters ( Records and Bytes written ) in Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1214) Add support for counters in
Hadoop Local Mode
Posted by "Ankit Modi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790119#action_12790119 ]
Ankit Modi commented on MAPREDUCE-1214:
---------------------------------------
Hi Jeff
I am trying to access counter values in code in pig. This is how it is done for Hadoop MR mode.
I am not able to replicate the process in Hadoop Local mode as we don't have an AssignedJobId.
{code}
RunningJob rj = jobClient.getJob( job.getAssignedJobID() );
Counters counters = rj.getCounters();
Counters.Group taskgroup = counters.getGroup("org.apache.hadoop.mapred.Task$Counter");
value = taskgroup.getCounterForName("REDUCE_OUTPUT_RECORDS").getCounter();
value2 = taskgroup.getCounterForName("MAP_OUTPUT_RECORDS").getCounter();
{code}
Is this the correct way, or am I missing something ?
> Add support for counters in Hadoop Local Mode
> ---------------------------------------------
>
> Key: MAPREDUCE-1214
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1214
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Ankit Modi
>
> Currently there is no support for counters ( Records and Bytes written ) in Hadoop Local Mode.
> Pig requires to provide counters to user when running in Hadoop Local Mode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.