You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Shi Yu <sh...@uchicago.edu> on 2011/06/07 18:21:37 UTC
Automatic line number in reducer output
Hi,
I am wondering is there any built-in function to automatically add a
self-increment line number in reducer output (like the relation DB
auto-key).
I have this problem because in 0.19.2 API, I used a variable linecount
increasing in the reducer like:
public static class Reduce extends MapReduceBase implements
Reducer<Text, IntWritable, Text,IntWritable>{
private long linecount = 0;
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output, Reporter reporter) throws
IOException {
//.....some code here
linecount ++;
output.collect(new Text(Long.toString(linecount)), var);
}
}
However, I found that this is not working in 0.20.2 API, if I write the
code like:
public static class Reduce extends
org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable>{
private long linecount = 0;
public void reduce (Text key, Iterator<IntWritable> values,
org.apache.hadoop.mapreduce.Reducer.Context context) throws IOException,
InterruptedException {
//some code here
linecount ++;
context.write(new Text(Long.toString(linecount)),var);
}
}
but it seems not working anymore.
I would also like to know if there are combiner and reducer implemented,
how to avoid that line number being written twice (cause I only want it
in reducer, not in combiner). Thanks!
Shi
Re: Automatic line number in reducer output
Posted by Shi Yu <sh...@uchicago.edu>.
Yes, it works perfectly. Actually didn't realize the flexibility to
employ different classes in combiner and reducer. In that case it would
have a three layer architecture, think that would be interesting and useful.
Shi
On 6/10/2011 9:31 AM, Robert Evans wrote:
> In this case you probably want two different classes. You can have the base Reducer class that adds in the line count, and then subclass it for the combiner, that sets a flag to not output the line numbers.
>
> --Bobby
>
>
> On 6/9/11 12:57 PM, "Shi Yu"<sh...@uchicago.edu> wrote:
>
> Hi,
>
> Thanks for the reply. The line count in new API works fine now, it was a
> bug in my code. In new API,
>
> Iterator is changed to Iterable,
>
> but I didn't pay attention to that and was still using Iterator and hasNext(), Next() method. Surprisingly, the wrong code still ran and got output, but the line number count did not work and I think it was null value. After fixing that Iterable mistake, the code works fine.
>
> The remaining problem is when combiner and reducer are both implemented, the output is like
>
> 00000 00000 value1
> 00001 00000 value2
> 00002 00000 value3
> 00003 00001 value4
> 00004 00001 value5
>
> The first column are counts from reducer, the second column are counts from combiner. I want to avoid the line counter in combiner, so my plan is to create another class which is almost the same as Reducer, but without the line count. I think it is doable to set Combiner and Reducer to different classes in jobconf, but I haven't tried it yet.
>
> Best,
>
> Shi
>
>
> On 6/9/2011 8:49 AM, Robert Evans wrote:
>
>> What exactly is linecount being output as in the new APIs?
>>
>> --Bobby
>>
>> On 6/7/11 11:21 AM, "Shi Yu"<sh...@uchicago.edu> wrote:
>>
>> Hi,
>>
>> I am wondering is there any built-in function to automatically add a
>> self-increment line number in reducer output (like the relation DB
>> auto-key).
>>
>> I have this problem because in 0.19.2 API, I used a variable linecount
>> increasing in the reducer like:
>>
>> public static class Reduce extends MapReduceBase implements
>> Reducer<Text, IntWritable, Text,IntWritable>{
>> private long linecount = 0;
>>
>> public void reduce(Text key, Iterator<IntWritable> values,
>> OutputCollector<Text, IntWritable> output, Reporter reporter) throws
>> IOException {
>>
>> //.....some code here
>> linecount ++;
>> output.collect(new Text(Long.toString(linecount)), var);
>>
>> }
>>
>> }
>>
>>
>> However, I found that this is not working in 0.20.2 API, if I write the
>> code like:
>>
>> public static class Reduce extends
>> org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable>{
>> private long linecount = 0;
>>
>> public void reduce (Text key, Iterator<IntWritable> values,
>> org.apache.hadoop.mapreduce.Reducer.Context context) throws IOException,
>> InterruptedException {
>>
>> //some code here
>> linecount ++;
>> context.write(new Text(Long.toString(linecount)),var);
>> }
>> }
>>
>> but it seems not working anymore.
>>
>>
>> I would also like to know if there are combiner and reducer implemented,
>> how to avoid that line number being written twice (cause I only want it
>> in reducer, not in combiner). Thanks!
>>
>>
>> Shi
>>
>>
>>
>>
>
>
--
Postdoctoral Scholar
Institute for Genomics and Systems Biology
Department of Medicine, the University of Chicago
Knapp Center for Biomedical Discovery
900 E. 57th St. Room 10148
Chicago, IL 60637, US
Tel: 773-702-6799
Re: Automatic line number in reducer output
Posted by Robert Evans <ev...@yahoo-inc.com>.
In this case you probably want two different classes. You can have the base Reducer class that adds in the line count, and then subclass it for the combiner, that sets a flag to not output the line numbers.
--Bobby
On 6/9/11 12:57 PM, "Shi Yu" <sh...@uchicago.edu> wrote:
Hi,
Thanks for the reply. The line count in new API works fine now, it was a
bug in my code. In new API,
Iterator is changed to Iterable,
but I didn't pay attention to that and was still using Iterator and hasNext(), Next() method. Surprisingly, the wrong code still ran and got output, but the line number count did not work and I think it was null value. After fixing that Iterable mistake, the code works fine.
The remaining problem is when combiner and reducer are both implemented, the output is like
00000 00000 value1
00001 00000 value2
00002 00000 value3
00003 00001 value4
00004 00001 value5
The first column are counts from reducer, the second column are counts from combiner. I want to avoid the line counter in combiner, so my plan is to create another class which is almost the same as Reducer, but without the line count. I think it is doable to set Combiner and Reducer to different classes in jobconf, but I haven't tried it yet.
Best,
Shi
On 6/9/2011 8:49 AM, Robert Evans wrote:
> What exactly is linecount being output as in the new APIs?
>
> --Bobby
>
> On 6/7/11 11:21 AM, "Shi Yu"<sh...@uchicago.edu> wrote:
>
> Hi,
>
> I am wondering is there any built-in function to automatically add a
> self-increment line number in reducer output (like the relation DB
> auto-key).
>
> I have this problem because in 0.19.2 API, I used a variable linecount
> increasing in the reducer like:
>
> public static class Reduce extends MapReduceBase implements
> Reducer<Text, IntWritable, Text,IntWritable>{
> private long linecount = 0;
>
> public void reduce(Text key, Iterator<IntWritable> values,
> OutputCollector<Text, IntWritable> output, Reporter reporter) throws
> IOException {
>
> //.....some code here
> linecount ++;
> output.collect(new Text(Long.toString(linecount)), var);
>
> }
>
> }
>
>
> However, I found that this is not working in 0.20.2 API, if I write the
> code like:
>
> public static class Reduce extends
> org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable>{
> private long linecount = 0;
>
> public void reduce (Text key, Iterator<IntWritable> values,
> org.apache.hadoop.mapreduce.Reducer.Context context) throws IOException,
> InterruptedException {
>
> //some code here
> linecount ++;
> context.write(new Text(Long.toString(linecount)),var);
> }
> }
>
> but it seems not working anymore.
>
>
> I would also like to know if there are combiner and reducer implemented,
> how to avoid that line number being written twice (cause I only want it
> in reducer, not in combiner). Thanks!
>
>
> Shi
>
>
>
>
Re: Automatic line number in reducer output
Posted by Shi Yu <sh...@uchicago.edu>.
Hi,
Thanks for the reply. The line count in new API works fine now, it was a
bug in my code. In new API,
Iterator is changed to Iterable,
but I didn't pay attention to that and was still using Iterator and hasNext(), Next() method. Surprisingly, the wrong code still ran and got output, but the line number count did not work and I think it was null value. After fixing that Iterable mistake, the code works fine.
The remaining problem is when combiner and reducer are both implemented, the output is like
00000 00000 value1
00001 00000 value2
00002 00000 value3
00003 00001 value4
00004 00001 value5
The first column are counts from reducer, the second column are counts from combiner. I want to avoid the line counter in combiner, so my plan is to create another class which is almost the same as Reducer, but without the line count. I think it is doable to set Combiner and Reducer to different classes in jobconf, but I haven't tried it yet.
Best,
Shi
On 6/9/2011 8:49 AM, Robert Evans wrote:
> What exactly is linecount being output as in the new APIs?
>
> --Bobby
>
> On 6/7/11 11:21 AM, "Shi Yu"<sh...@uchicago.edu> wrote:
>
> Hi,
>
> I am wondering is there any built-in function to automatically add a
> self-increment line number in reducer output (like the relation DB
> auto-key).
>
> I have this problem because in 0.19.2 API, I used a variable linecount
> increasing in the reducer like:
>
> public static class Reduce extends MapReduceBase implements
> Reducer<Text, IntWritable, Text,IntWritable>{
> private long linecount = 0;
>
> public void reduce(Text key, Iterator<IntWritable> values,
> OutputCollector<Text, IntWritable> output, Reporter reporter) throws
> IOException {
>
> //.....some code here
> linecount ++;
> output.collect(new Text(Long.toString(linecount)), var);
>
> }
>
> }
>
>
> However, I found that this is not working in 0.20.2 API, if I write the
> code like:
>
> public static class Reduce extends
> org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable>{
> private long linecount = 0;
>
> public void reduce (Text key, Iterator<IntWritable> values,
> org.apache.hadoop.mapreduce.Reducer.Context context) throws IOException,
> InterruptedException {
>
> //some code here
> linecount ++;
> context.write(new Text(Long.toString(linecount)),var);
> }
> }
>
> but it seems not working anymore.
>
>
> I would also like to know if there are combiner and reducer implemented,
> how to avoid that line number being written twice (cause I only want it
> in reducer, not in combiner). Thanks!
>
>
> Shi
>
>
>
>
Re: Automatic line number in reducer output
Posted by Robert Evans <ev...@yahoo-inc.com>.
What exactly is linecount being output as in the new APIs?
--Bobby
On 6/7/11 11:21 AM, "Shi Yu" <sh...@uchicago.edu> wrote:
Hi,
I am wondering is there any built-in function to automatically add a
self-increment line number in reducer output (like the relation DB
auto-key).
I have this problem because in 0.19.2 API, I used a variable linecount
increasing in the reducer like:
public static class Reduce extends MapReduceBase implements
Reducer<Text, IntWritable, Text,IntWritable>{
private long linecount = 0;
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output, Reporter reporter) throws
IOException {
//.....some code here
linecount ++;
output.collect(new Text(Long.toString(linecount)), var);
}
}
However, I found that this is not working in 0.20.2 API, if I write the
code like:
public static class Reduce extends
org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable>{
private long linecount = 0;
public void reduce (Text key, Iterator<IntWritable> values,
org.apache.hadoop.mapreduce.Reducer.Context context) throws IOException,
InterruptedException {
//some code here
linecount ++;
context.write(new Text(Long.toString(linecount)),var);
}
}
but it seems not working anymore.
I would also like to know if there are combiner and reducer implemented,
how to avoid that line number being written twice (cause I only want it
in reducer, not in combiner). Thanks!
Shi