You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Reza <pd...@eden.rutgers.edu> on 2009/04/18 19:21:19 UTC
Help with Map Reduce
Hello all:
I am new to Hadoop and Map Reduce. I am writing a program to analyze some
census data.
I have a general question with MapReduce:
In the Reducer, how can I separate keys to do separate calculations based
on the key? In my case, I am trying to use if statements to separate the
keys out, but for some reason, it is not doing so.
Here is a segment of my code. My mapper seems to work, I think it is my
Reducer that is messing up:
public static class MapClass extends MapReduceBase implements
Mapper<LongWritable, Text, Text, IntWritable> {
// private final static IntWritable one = new IntWritable(1);
Text sex = new Text("sex");
public void map(LongWritable key, Text value,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer itr = new StringTokenizer(line,",");
:
:
word.set(itr.nextToken());
IntWritable sexVal = new
IntWritable(Integer.parseInt(word.toString())); //map gender
output.collect(sex, sexVal);
}//end map()method
}//end class MapClass
public static class Reduce extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
int totalPop = 0; //count: total population
int numMales = 0; //count: males
int numFemales = 0; //count: females
Text popOut = new Text("The total population is: ");
if (key.equals("sex")) {
while (values.hasNext()) {
if (values.next().get() == 0)
numMales++; //number of males
else
numFemales++; //number of
females
}//end while
totalPop = numMales + numFemales; //count of population
}//end if
output.collect(popOut, new IntWritable(totalPop));
}//end reduce method
}//end class Reduce
key is of type Text, I also used: if (key.toString() == "sex) {....
but that wasnt working either.
There are of course other keys and values, but I would think to gather all
the keys "sex" would be done with the way I show it above. However, My
output reads as 0. I tried changing the numMales value to see if the if
statement is executed, but it is not. Am I doing something wrong?
Re: Help with Map Reduce
Posted by Tushar Jain <tu...@gmail.com>.
s/to things/two things/ ;)
Tushar
On Sun, Apr 19, 2009 at 6:20 PM, Tushar Jain <tu...@gmail.com> wrote:
> Could you try to things, just to further test your theory that the
> equals is whats failing:
>
> 1. Log the value of the key (so the result of key.toString()), just to
> see what it is.
> 2. Do as below:
>
> Test sexKey = new Text("sex");
>
> if(key.equals(sexKey)) {
> .
> .
> }
>
> These 2 should atleast help confirm or deny your theory.
>
> Hope this helps
>
> Tushar
>
>
>
> On Sun, Apr 19, 2009 at 9:54 AM, Reza <pd...@eden.rutgers.edu> wrote:
>> Hey Brian:
>>
>> I tried that as well, but to no avail . =[.
>>
>> -Reza
>>
>>> Hm,
>>>
>>> I don't know how equals() is implemented for Text, but I'd try:
>>>
>>> key.toString().equals("sex")
>>>
>>> Brian
>>>
>>> On Apr 19, 2009, at 11:29 AM, Reza wrote:
>>>
>>>> Brian:
>>>>
>>>> Thanks for your response.
>>>>
>>>> I have 8 total keys and values. The code I show below is part of the
>>>> whole
>>>> thing, just to illustrate my problem.
>>>>
>>>> Im trying to call each key with an if statement (as in the piece I
>>>> showed
>>>> before). Each of my 8 keys have their respective if statement.
>>>>
>>>> However, it seems that my if statement is not going through. I even
>>>> put
>>>> the output.collect(...) inside the if statement, and accordingly, no
>>>> output was shown. I already tested my mapper, and it seems to work
>>>> fine.
>>>>
>>>> I believe that I am having difficulty at this point:
>>>>
>>>> if (key.equals("sex")) {...}
>>>>
>>>> The key in this case is "sex" (or gender) and the possible values
>>>> are 0
>>>> (for male) and 1 (for female). Is this not the right way to separate
>>>> out
>>>> my keys?
>>>>
>>>> -Reza
>>>>
>>>>> Hey Reza,
>>>>>
>>>>> From reading your code, you are calling this for the key "sex":
>>>>>
>>>>> output.collect("The total population is: ", (actual population))
>>>>>
>>>>> and, for every other key:
>>>>>
>>>>> output.collect("The total population is: ", 0)
>>>>>
>>>>> You probably only want to call the output collector in the first
>>>>> case,
>>>>> not every time.
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>> On Apr 18, 2009, at 12:21 PM, Reza wrote:
>>>>>
>>>>>> Hello all:
>>>>>>
>>>>>> I am new to Hadoop and Map Reduce. I am writing a program to analyze
>>>>>> some
>>>>>> census data.
>>>>>>
>>>>>> I have a general question with MapReduce:
>>>>>>
>>>>>> In the Reducer, how can I separate keys to do separate calculations
>>>>>> based
>>>>>> on the key? In my case, I am trying to use if statements to separate
>>>>>> the
>>>>>> keys out, but for some reason, it is not doing so.
>>>>>>
>>>>>> Here is a segment of my code. My mapper seems to work, I think it is
>>>>>> my
>>>>>> Reducer that is messing up:
>>>>>>
>>>>>> public static class MapClass extends MapReduceBase implements
>>>>>> Mapper<LongWritable, Text, Text, IntWritable> {
>>>>>>
>>>>>> // private final static IntWritable one = new IntWritable(1);
>>>>>> Text sex = new Text("sex");
>>>>>> public void map(LongWritable key, Text value,
>>>>>> OutputCollector<Text, IntWritable> output,
>>>>>> Reporter reporter) throws IOException {
>>>>>> String line = value.toString();
>>>>>> StringTokenizer itr = new StringTokenizer(line,",");
>>>>>>
>>>>>> :
>>>>>> :
>>>>>> word.set(itr.nextToken());
>>>>>> IntWritable sexVal = new
>>>>>> IntWritable(Integer.parseInt(word.toString())); //map gender
>>>>>> output.collect(sex, sexVal);
>>>>>>
>>>>>> }//end map()method
>>>>>> }//end class MapClass
>>>>>>
>>>>>>
>>>>>> public static class Reduce extends MapReduceBase
>>>>>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>>>>>>
>>>>>>
>>>>>> public void reduce(Text key, Iterator<IntWritable> values,
>>>>>> OutputCollector<Text, IntWritable> output,
>>>>>> Reporter reporter) throws IOException {
>>>>>>
>>>>>> int totalPop = 0; //count: total population
>>>>>> int numMales = 0; //count: males
>>>>>> int numFemales = 0; //count: females
>>>>>> Text popOut = new Text("The total population is:
>>>>>> ");
>>>>>> if (key.equals("sex")) {
>>>>>> while (values.hasNext()) {
>>>>>> if (values.next().get() == 0)
>>>>>> numMales++; //number
>>>>>> of males
>>>>>> else
>>>>>> numFemales++; //number
>>>>>> of
>>>>>> females
>>>>>> }//end while
>>>>>> totalPop = numMales + numFemales; //count of
>>>>>> population
>>>>>> }//end if
>>>>>> output.collect(popOut, new IntWritable(totalPop));
>>>>>>
>>>>>> }//end reduce method
>>>>>> }//end class Reduce
>>>>>>
>>>>>> key is of type Text, I also used: if (key.toString() == "sex) {....
>>>>>> but that wasnt working either.
>>>>>>
>>>>>> There are of course other keys and values, but I would think to
>>>>>> gather all
>>>>>> the keys "sex" would be done with the way I show it above. However,
>>>>>> My
>>>>>> output reads as 0. I tried changing the numMales value to see if
>>>>>> the if
>>>>>> statement is executed, but it is not. Am I doing something wrong?
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>>
>
Re: Help with Map Reduce
Posted by Reza <pd...@eden.rutgers.edu>.
Tushar:
I also tried that to make sure I wasnt going crazy, but it still did not
go through the if statement. I do not know why it is bypassing it. This is
very weird to me. Unless I have missed something, the logic should be
working.
Reza
> Could you try to things, just to further test your theory that the
> equals is whats failing:
>
> 1. Log the value of the key (so the result of key.toString()), just to
> see what it is.
> 2. Do as below:
>
> Test sexKey = new Text("sex");
>
> if(key.equals(sexKey)) {
> .
> .
> }
>
> These 2 should atleast help confirm or deny your theory.
>
> Hope this helps
>
> Tushar
>
>
>
> On Sun, Apr 19, 2009 at 9:54 AM, Reza <pd...@eden.rutgers.edu> wrote:
>> Hey Brian:
>>
>> I tried that as well, but to no avail . =[.
>>
>> -Reza
>>
>>> Hm,
>>>
>>> I don't know how equals() is implemented for Text, but I'd try:
>>>
>>> key.toString().equals("sex")
>>>
>>> Brian
>>>
>>> On Apr 19, 2009, at 11:29 AM, Reza wrote:
>>>
>>>> Brian:
>>>>
>>>> Thanks for your response.
>>>>
>>>> I have 8 total keys and values. The code I show below is part of the
>>>> whole
>>>> thing, just to illustrate my problem.
>>>>
>>>> Im trying to call each key with an if statement (as in the piece I
>>>> showed
>>>> before). Each of my 8 keys have their respective if statement.
>>>>
>>>> However, it seems that my if statement is not going through. I even
>>>> put
>>>> the output.collect(...) inside the if statement, and accordingly, no
>>>> output was shown. I already tested my mapper, and it seems to work
>>>> fine.
>>>>
>>>> I believe that I am having difficulty at this point:
>>>>
>>>> if (key.equals("sex")) {...}
>>>>
>>>> The key in this case is "sex" (or gender) and the possible values
>>>> are 0
>>>> (for male) and 1 (for female). Is this not the right way to separate
>>>> out
>>>> my keys?
>>>>
>>>> -Reza
>>>>
>>>>> Hey Reza,
>>>>>
>>>>> From reading your code, you are calling this for the key "sex":
>>>>>
>>>>> output.collect("The total population is: ", (actual population))
>>>>>
>>>>> and, for every other key:
>>>>>
>>>>> output.collect("The total population is: ", 0)
>>>>>
>>>>> You probably only want to call the output collector in the first
>>>>> case,
>>>>> not every time.
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>> On Apr 18, 2009, at 12:21 PM, Reza wrote:
>>>>>
>>>>>> Hello all:
>>>>>>
>>>>>> I am new to Hadoop and Map Reduce. I am writing a program to analyze
>>>>>> some
>>>>>> census data.
>>>>>>
>>>>>> I have a general question with MapReduce:
>>>>>>
>>>>>> In the Reducer, how can I separate keys to do separate calculations
>>>>>> based
>>>>>> on the key? In my case, I am trying to use if statements to separate
>>>>>> the
>>>>>> keys out, but for some reason, it is not doing so.
>>>>>>
>>>>>> Here is a segment of my code. My mapper seems to work, I think it is
>>>>>> my
>>>>>> Reducer that is messing up:
>>>>>>
>>>>>> public static class MapClass extends MapReduceBase implements
>>>>>> Mapper<LongWritable, Text, Text, IntWritable> {
>>>>>>
>>>>>> // private final static IntWritable one = new IntWritable(1);
>>>>>> Text sex = new Text("sex");
>>>>>> public void map(LongWritable key, Text value,
>>>>>> OutputCollector<Text, IntWritable> output,
>>>>>> Reporter reporter) throws IOException {
>>>>>> String line = value.toString();
>>>>>> StringTokenizer itr = new StringTokenizer(line,",");
>>>>>>
>>>>>> :
>>>>>> :
>>>>>> word.set(itr.nextToken());
>>>>>> IntWritable sexVal = new
>>>>>> IntWritable(Integer.parseInt(word.toString())); //map gender
>>>>>> output.collect(sex, sexVal);
>>>>>>
>>>>>> }//end map()method
>>>>>> }//end class MapClass
>>>>>>
>>>>>>
>>>>>> public static class Reduce extends MapReduceBase
>>>>>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>>>>>>
>>>>>>
>>>>>> public void reduce(Text key, Iterator<IntWritable> values,
>>>>>> OutputCollector<Text, IntWritable> output,
>>>>>> Reporter reporter) throws IOException {
>>>>>>
>>>>>> int totalPop = 0; //count: total population
>>>>>> int numMales = 0; //count: males
>>>>>> int numFemales = 0; //count: females
>>>>>> Text popOut = new Text("The total population is:
>>>>>> ");
>>>>>> if (key.equals("sex")) {
>>>>>> while (values.hasNext()) {
>>>>>> if (values.next().get() == 0)
>>>>>> numMales++; //number
>>>>>> of males
>>>>>> else
>>>>>> numFemales++; //number
>>>>>> of
>>>>>> females
>>>>>> }//end while
>>>>>> totalPop = numMales + numFemales; //count of
>>>>>> population
>>>>>> }//end if
>>>>>> output.collect(popOut, new IntWritable(totalPop));
>>>>>>
>>>>>> }//end reduce method
>>>>>> }//end class Reduce
>>>>>>
>>>>>> key is of type Text, I also used: if (key.toString() == "sex) {....
>>>>>> but that wasnt working either.
>>>>>>
>>>>>> There are of course other keys and values, but I would think to
>>>>>> gather all
>>>>>> the keys "sex" would be done with the way I show it above. However,
>>>>>> My
>>>>>> output reads as 0. I tried changing the numMales value to see if
>>>>>> the if
>>>>>> statement is executed, but it is not. Am I doing something wrong?
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>>
>
Re: Help with Map Reduce
Posted by Tushar Jain <tu...@gmail.com>.
Could you try to things, just to further test your theory that the
equals is whats failing:
1. Log the value of the key (so the result of key.toString()), just to
see what it is.
2. Do as below:
Test sexKey = new Text("sex");
if(key.equals(sexKey)) {
.
.
}
These 2 should atleast help confirm or deny your theory.
Hope this helps
Tushar
On Sun, Apr 19, 2009 at 9:54 AM, Reza <pd...@eden.rutgers.edu> wrote:
> Hey Brian:
>
> I tried that as well, but to no avail . =[.
>
> -Reza
>
>> Hm,
>>
>> I don't know how equals() is implemented for Text, but I'd try:
>>
>> key.toString().equals("sex")
>>
>> Brian
>>
>> On Apr 19, 2009, at 11:29 AM, Reza wrote:
>>
>>> Brian:
>>>
>>> Thanks for your response.
>>>
>>> I have 8 total keys and values. The code I show below is part of the
>>> whole
>>> thing, just to illustrate my problem.
>>>
>>> Im trying to call each key with an if statement (as in the piece I
>>> showed
>>> before). Each of my 8 keys have their respective if statement.
>>>
>>> However, it seems that my if statement is not going through. I even
>>> put
>>> the output.collect(...) inside the if statement, and accordingly, no
>>> output was shown. I already tested my mapper, and it seems to work
>>> fine.
>>>
>>> I believe that I am having difficulty at this point:
>>>
>>> if (key.equals("sex")) {...}
>>>
>>> The key in this case is "sex" (or gender) and the possible values
>>> are 0
>>> (for male) and 1 (for female). Is this not the right way to separate
>>> out
>>> my keys?
>>>
>>> -Reza
>>>
>>>> Hey Reza,
>>>>
>>>> From reading your code, you are calling this for the key "sex":
>>>>
>>>> output.collect("The total population is: ", (actual population))
>>>>
>>>> and, for every other key:
>>>>
>>>> output.collect("The total population is: ", 0)
>>>>
>>>> You probably only want to call the output collector in the first
>>>> case,
>>>> not every time.
>>>>
>>>> Brian
>>>>
>>>>
>>>> On Apr 18, 2009, at 12:21 PM, Reza wrote:
>>>>
>>>>> Hello all:
>>>>>
>>>>> I am new to Hadoop and Map Reduce. I am writing a program to analyze
>>>>> some
>>>>> census data.
>>>>>
>>>>> I have a general question with MapReduce:
>>>>>
>>>>> In the Reducer, how can I separate keys to do separate calculations
>>>>> based
>>>>> on the key? In my case, I am trying to use if statements to separate
>>>>> the
>>>>> keys out, but for some reason, it is not doing so.
>>>>>
>>>>> Here is a segment of my code. My mapper seems to work, I think it is
>>>>> my
>>>>> Reducer that is messing up:
>>>>>
>>>>> public static class MapClass extends MapReduceBase implements
>>>>> Mapper<LongWritable, Text, Text, IntWritable> {
>>>>>
>>>>> // private final static IntWritable one = new IntWritable(1);
>>>>> Text sex = new Text("sex");
>>>>> public void map(LongWritable key, Text value,
>>>>> OutputCollector<Text, IntWritable> output,
>>>>> Reporter reporter) throws IOException {
>>>>> String line = value.toString();
>>>>> StringTokenizer itr = new StringTokenizer(line,",");
>>>>>
>>>>> :
>>>>> :
>>>>> word.set(itr.nextToken());
>>>>> IntWritable sexVal = new
>>>>> IntWritable(Integer.parseInt(word.toString())); //map gender
>>>>> output.collect(sex, sexVal);
>>>>>
>>>>> }//end map()method
>>>>> }//end class MapClass
>>>>>
>>>>>
>>>>> public static class Reduce extends MapReduceBase
>>>>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>>>>>
>>>>>
>>>>> public void reduce(Text key, Iterator<IntWritable> values,
>>>>> OutputCollector<Text, IntWritable> output,
>>>>> Reporter reporter) throws IOException {
>>>>>
>>>>> int totalPop = 0; //count: total population
>>>>> int numMales = 0; //count: males
>>>>> int numFemales = 0; //count: females
>>>>> Text popOut = new Text("The total population is:
>>>>> ");
>>>>> if (key.equals("sex")) {
>>>>> while (values.hasNext()) {
>>>>> if (values.next().get() == 0)
>>>>> numMales++; //number
>>>>> of males
>>>>> else
>>>>> numFemales++; //number
>>>>> of
>>>>> females
>>>>> }//end while
>>>>> totalPop = numMales + numFemales; //count of
>>>>> population
>>>>> }//end if
>>>>> output.collect(popOut, new IntWritable(totalPop));
>>>>>
>>>>> }//end reduce method
>>>>> }//end class Reduce
>>>>>
>>>>> key is of type Text, I also used: if (key.toString() == "sex) {....
>>>>> but that wasnt working either.
>>>>>
>>>>> There are of course other keys and values, but I would think to
>>>>> gather all
>>>>> the keys "sex" would be done with the way I show it above. However,
>>>>> My
>>>>> output reads as 0. I tried changing the numMales value to see if
>>>>> the if
>>>>> statement is executed, but it is not. Am I doing something wrong?
>>>>>
>>>>
>>>>
>>
>>
>
>
Re: Help with Map Reduce
Posted by Reza <pd...@eden.rutgers.edu>.
Hey Brian:
I tried that as well, but to no avail . =[.
-Reza
> Hm,
>
> I don't know how equals() is implemented for Text, but I'd try:
>
> key.toString().equals("sex")
>
> Brian
>
> On Apr 19, 2009, at 11:29 AM, Reza wrote:
>
>> Brian:
>>
>> Thanks for your response.
>>
>> I have 8 total keys and values. The code I show below is part of the
>> whole
>> thing, just to illustrate my problem.
>>
>> Im trying to call each key with an if statement (as in the piece I
>> showed
>> before). Each of my 8 keys have their respective if statement.
>>
>> However, it seems that my if statement is not going through. I even
>> put
>> the output.collect(...) inside the if statement, and accordingly, no
>> output was shown. I already tested my mapper, and it seems to work
>> fine.
>>
>> I believe that I am having difficulty at this point:
>>
>> if (key.equals("sex")) {...}
>>
>> The key in this case is "sex" (or gender) and the possible values
>> are 0
>> (for male) and 1 (for female). Is this not the right way to separate
>> out
>> my keys?
>>
>> -Reza
>>
>>> Hey Reza,
>>>
>>> From reading your code, you are calling this for the key "sex":
>>>
>>> output.collect("The total population is: ", (actual population))
>>>
>>> and, for every other key:
>>>
>>> output.collect("The total population is: ", 0)
>>>
>>> You probably only want to call the output collector in the first
>>> case,
>>> not every time.
>>>
>>> Brian
>>>
>>>
>>> On Apr 18, 2009, at 12:21 PM, Reza wrote:
>>>
>>>> Hello all:
>>>>
>>>> I am new to Hadoop and Map Reduce. I am writing a program to analyze
>>>> some
>>>> census data.
>>>>
>>>> I have a general question with MapReduce:
>>>>
>>>> In the Reducer, how can I separate keys to do separate calculations
>>>> based
>>>> on the key? In my case, I am trying to use if statements to separate
>>>> the
>>>> keys out, but for some reason, it is not doing so.
>>>>
>>>> Here is a segment of my code. My mapper seems to work, I think it is
>>>> my
>>>> Reducer that is messing up:
>>>>
>>>> public static class MapClass extends MapReduceBase implements
>>>> Mapper<LongWritable, Text, Text, IntWritable> {
>>>>
>>>> // private final static IntWritable one = new IntWritable(1);
>>>> Text sex = new Text("sex");
>>>> public void map(LongWritable key, Text value,
>>>> OutputCollector<Text, IntWritable> output,
>>>> Reporter reporter) throws IOException {
>>>> String line = value.toString();
>>>> StringTokenizer itr = new StringTokenizer(line,",");
>>>>
>>>> :
>>>> :
>>>> word.set(itr.nextToken());
>>>> IntWritable sexVal = new
>>>> IntWritable(Integer.parseInt(word.toString())); //map gender
>>>> output.collect(sex, sexVal);
>>>>
>>>> }//end map()method
>>>> }//end class MapClass
>>>>
>>>>
>>>> public static class Reduce extends MapReduceBase
>>>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>>>>
>>>>
>>>> public void reduce(Text key, Iterator<IntWritable> values,
>>>> OutputCollector<Text, IntWritable> output,
>>>> Reporter reporter) throws IOException {
>>>>
>>>> int totalPop = 0; //count: total population
>>>> int numMales = 0; //count: males
>>>> int numFemales = 0; //count: females
>>>> Text popOut = new Text("The total population is:
>>>> ");
>>>> if (key.equals("sex")) {
>>>> while (values.hasNext()) {
>>>> if (values.next().get() == 0)
>>>> numMales++; //number
>>>> of males
>>>> else
>>>> numFemales++; //number
>>>> of
>>>> females
>>>> }//end while
>>>> totalPop = numMales + numFemales; //count of
>>>> population
>>>> }//end if
>>>> output.collect(popOut, new IntWritable(totalPop));
>>>>
>>>> }//end reduce method
>>>> }//end class Reduce
>>>>
>>>> key is of type Text, I also used: if (key.toString() == "sex) {....
>>>> but that wasnt working either.
>>>>
>>>> There are of course other keys and values, but I would think to
>>>> gather all
>>>> the keys "sex" would be done with the way I show it above. However,
>>>> My
>>>> output reads as 0. I tried changing the numMales value to see if
>>>> the if
>>>> statement is executed, but it is not. Am I doing something wrong?
>>>>
>>>
>>>
>
>
Re: Help with Map Reduce
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hm,
I don't know how equals() is implemented for Text, but I'd try:
key.toString().equals("sex")
Brian
On Apr 19, 2009, at 11:29 AM, Reza wrote:
> Brian:
>
> Thanks for your response.
>
> I have 8 total keys and values. The code I show below is part of the
> whole
> thing, just to illustrate my problem.
>
> Im trying to call each key with an if statement (as in the piece I
> showed
> before). Each of my 8 keys have their respective if statement.
>
> However, it seems that my if statement is not going through. I even
> put
> the output.collect(...) inside the if statement, and accordingly, no
> output was shown. I already tested my mapper, and it seems to work
> fine.
>
> I believe that I am having difficulty at this point:
>
> if (key.equals("sex")) {...}
>
> The key in this case is "sex" (or gender) and the possible values
> are 0
> (for male) and 1 (for female). Is this not the right way to separate
> out
> my keys?
>
> -Reza
>
>> Hey Reza,
>>
>> From reading your code, you are calling this for the key "sex":
>>
>> output.collect("The total population is: ", (actual population))
>>
>> and, for every other key:
>>
>> output.collect("The total population is: ", 0)
>>
>> You probably only want to call the output collector in the first
>> case,
>> not every time.
>>
>> Brian
>>
>>
>> On Apr 18, 2009, at 12:21 PM, Reza wrote:
>>
>>> Hello all:
>>>
>>> I am new to Hadoop and Map Reduce. I am writing a program to analyze
>>> some
>>> census data.
>>>
>>> I have a general question with MapReduce:
>>>
>>> In the Reducer, how can I separate keys to do separate calculations
>>> based
>>> on the key? In my case, I am trying to use if statements to separate
>>> the
>>> keys out, but for some reason, it is not doing so.
>>>
>>> Here is a segment of my code. My mapper seems to work, I think it is
>>> my
>>> Reducer that is messing up:
>>>
>>> public static class MapClass extends MapReduceBase implements
>>> Mapper<LongWritable, Text, Text, IntWritable> {
>>>
>>> // private final static IntWritable one = new IntWritable(1);
>>> Text sex = new Text("sex");
>>> public void map(LongWritable key, Text value,
>>> OutputCollector<Text, IntWritable> output,
>>> Reporter reporter) throws IOException {
>>> String line = value.toString();
>>> StringTokenizer itr = new StringTokenizer(line,",");
>>>
>>> :
>>> :
>>> word.set(itr.nextToken());
>>> IntWritable sexVal = new
>>> IntWritable(Integer.parseInt(word.toString())); //map gender
>>> output.collect(sex, sexVal);
>>>
>>> }//end map()method
>>> }//end class MapClass
>>>
>>>
>>> public static class Reduce extends MapReduceBase
>>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>>>
>>>
>>> public void reduce(Text key, Iterator<IntWritable> values,
>>> OutputCollector<Text, IntWritable> output,
>>> Reporter reporter) throws IOException {
>>>
>>> int totalPop = 0; //count: total population
>>> int numMales = 0; //count: males
>>> int numFemales = 0; //count: females
>>> Text popOut = new Text("The total population is:
>>> ");
>>> if (key.equals("sex")) {
>>> while (values.hasNext()) {
>>> if (values.next().get() == 0)
>>> numMales++; //number
>>> of males
>>> else
>>> numFemales++; //number
>>> of
>>> females
>>> }//end while
>>> totalPop = numMales + numFemales; //count of
>>> population
>>> }//end if
>>> output.collect(popOut, new IntWritable(totalPop));
>>>
>>> }//end reduce method
>>> }//end class Reduce
>>>
>>> key is of type Text, I also used: if (key.toString() == "sex) {....
>>> but that wasnt working either.
>>>
>>> There are of course other keys and values, but I would think to
>>> gather all
>>> the keys "sex" would be done with the way I show it above. However,
>>> My
>>> output reads as 0. I tried changing the numMales value to see if
>>> the if
>>> statement is executed, but it is not. Am I doing something wrong?
>>>
>>
>>
Re: Help with Map Reduce
Posted by Reza <pd...@eden.rutgers.edu>.
Brian:
Thanks for your response.
I have 8 total keys and values. The code I show below is part of the whole
thing, just to illustrate my problem.
Im trying to call each key with an if statement (as in the piece I showed
before). Each of my 8 keys have their respective if statement.
However, it seems that my if statement is not going through. I even put
the output.collect(...) inside the if statement, and accordingly, no
output was shown. I already tested my mapper, and it seems to work fine.
I believe that I am having difficulty at this point:
if (key.equals("sex")) {...}
The key in this case is "sex" (or gender) and the possible values are 0
(for male) and 1 (for female). Is this not the right way to separate out
my keys?
-Reza
> Hey Reza,
>
> From reading your code, you are calling this for the key "sex":
>
> output.collect("The total population is: ", (actual population))
>
> and, for every other key:
>
> output.collect("The total population is: ", 0)
>
> You probably only want to call the output collector in the first case,
> not every time.
>
> Brian
>
>
> On Apr 18, 2009, at 12:21 PM, Reza wrote:
>
>> Hello all:
>>
>> I am new to Hadoop and Map Reduce. I am writing a program to analyze
>> some
>> census data.
>>
>> I have a general question with MapReduce:
>>
>> In the Reducer, how can I separate keys to do separate calculations
>> based
>> on the key? In my case, I am trying to use if statements to separate
>> the
>> keys out, but for some reason, it is not doing so.
>>
>> Here is a segment of my code. My mapper seems to work, I think it is
>> my
>> Reducer that is messing up:
>>
>> public static class MapClass extends MapReduceBase implements
>> Mapper<LongWritable, Text, Text, IntWritable> {
>>
>> // private final static IntWritable one = new IntWritable(1);
>> Text sex = new Text("sex");
>> public void map(LongWritable key, Text value,
>> OutputCollector<Text, IntWritable> output,
>> Reporter reporter) throws IOException {
>> String line = value.toString();
>> StringTokenizer itr = new StringTokenizer(line,",");
>>
>> :
>> :
>> word.set(itr.nextToken());
>> IntWritable sexVal = new
>> IntWritable(Integer.parseInt(word.toString())); //map gender
>> output.collect(sex, sexVal);
>>
>> }//end map()method
>> }//end class MapClass
>>
>>
>> public static class Reduce extends MapReduceBase
>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>>
>>
>> public void reduce(Text key, Iterator<IntWritable> values,
>> OutputCollector<Text, IntWritable> output,
>> Reporter reporter) throws IOException {
>>
>> int totalPop = 0; //count: total population
>> int numMales = 0; //count: males
>> int numFemales = 0; //count: females
>> Text popOut = new Text("The total population is:
>> ");
>> if (key.equals("sex")) {
>> while (values.hasNext()) {
>> if (values.next().get() == 0)
>> numMales++; //number
>> of males
>> else
>> numFemales++; //number
>> of
>> females
>> }//end while
>> totalPop = numMales + numFemales; //count of population
>> }//end if
>> output.collect(popOut, new IntWritable(totalPop));
>>
>> }//end reduce method
>> }//end class Reduce
>>
>> key is of type Text, I also used: if (key.toString() == "sex) {....
>> but that wasnt working either.
>>
>> There are of course other keys and values, but I would think to
>> gather all
>> the keys "sex" would be done with the way I show it above. However,
>> My
>> output reads as 0. I tried changing the numMales value to see if
>> the if
>> statement is executed, but it is not. Am I doing something wrong?
>>
>
>
Re: Help with Map Reduce
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hey Reza,
From reading your code, you are calling this for the key "sex":
output.collect("The total population is: ", (actual population))
and, for every other key:
output.collect("The total population is: ", 0)
You probably only want to call the output collector in the first case,
not every time.
Brian
On Apr 18, 2009, at 12:21 PM, Reza wrote:
> Hello all:
>
> I am new to Hadoop and Map Reduce. I am writing a program to analyze
> some
> census data.
>
> I have a general question with MapReduce:
>
> In the Reducer, how can I separate keys to do separate calculations
> based
> on the key? In my case, I am trying to use if statements to separate
> the
> keys out, but for some reason, it is not doing so.
>
> Here is a segment of my code. My mapper seems to work, I think it is
> my
> Reducer that is messing up:
>
> public static class MapClass extends MapReduceBase implements
> Mapper<LongWritable, Text, Text, IntWritable> {
>
> // private final static IntWritable one = new IntWritable(1);
> Text sex = new Text("sex");
> public void map(LongWritable key, Text value,
> OutputCollector<Text, IntWritable> output,
> Reporter reporter) throws IOException {
> String line = value.toString();
> StringTokenizer itr = new StringTokenizer(line,",");
>
> :
> :
> word.set(itr.nextToken());
> IntWritable sexVal = new
> IntWritable(Integer.parseInt(word.toString())); //map gender
> output.collect(sex, sexVal);
>
> }//end map()method
> }//end class MapClass
>
>
> public static class Reduce extends MapReduceBase
> implements Reducer<Text, IntWritable, Text, IntWritable> {
>
>
> public void reduce(Text key, Iterator<IntWritable> values,
> OutputCollector<Text, IntWritable> output,
> Reporter reporter) throws IOException {
>
> int totalPop = 0; //count: total population
> int numMales = 0; //count: males
> int numFemales = 0; //count: females
> Text popOut = new Text("The total population is:
> ");
> if (key.equals("sex")) {
> while (values.hasNext()) {
> if (values.next().get() == 0)
> numMales++; //number
> of males
> else
> numFemales++; //number
> of
> females
> }//end while
> totalPop = numMales + numFemales; //count of population
> }//end if
> output.collect(popOut, new IntWritable(totalPop));
>
> }//end reduce method
> }//end class Reduce
>
> key is of type Text, I also used: if (key.toString() == "sex) {....
> but that wasnt working either.
>
> There are of course other keys and values, but I would think to
> gather all
> the keys "sex" would be done with the way I show it above. However,
> My
> output reads as 0. I tried changing the numMales value to see if
> the if
> statement is executed, but it is not. Am I doing something wrong?
>