You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Deepika Khera <De...@avg.com> on 2010/05/24 23:35:10 UTC
Hash Partitioner
Hi,
I am using a HashPartitioner on my key for a map reducer job. I am wondering how sometimes 2 reducers end up getting the same key ? I have the hashCode method defined for my key.
Also, I have speculative execution turned off for my jobs..
Would appreciate any help.
Thanks,
Deepika
RE: RE: Re: Re: Hash Partitioner
Posted by Deepika Khera <De...@avg.com>.
This is just to close this one. I finally resolved my issue. Problem was that I had some enums in my key, the hash code of which is not constant over JVMs. So instead of doing myEnum.hashCode(), I should have first converted it to a string and then taken the hashcode (like myEnum.name().hashCode()). I was relying on a correct hashCode() method, since the IDE had generated it for me. I just had to be careful for my case.
Thanks for all the help!
Deepika
-----Original Message-----
From: Deepika Khera [mailto:Deepika.Khera@avg.com]
Sent: Tuesday, May 25, 2010 2:03 PM
To: general@hadoop.apache.org
Subject: RE: Re: Re: Hash Partitioner
So I ran my process again with some more logging and here is what I see.
I used my own HashPartitioner(basically copied hadoop's partitioner and added some logging for analysis).I printed here the Key and the reducer that is assigned to the key (based on the hash code).
My process triggered off 2 mappers (running on 2 different hadoop machines), hence both of these try to find reducers for the split file assigned to them. I see that for a same object key assigned to both these mappers, I am getting 2 different reducers allocated by the Partitioner.
In the reducers I see -
1) 2 different reducers (the ones that the partitioner assigned the key to) printing out the same Key (I did not print out the value as I thought that wouldn't matter)
2) Here are the logs from where reducers copies data from the mapper -
Reducer1:
2010-05-25 11:34:49,810 INFO org.apache.hadoop.mapred.ReduceTask: Read 1002612 bytes from map-output for attempt_201005251129_0001_m_000001_0
2010-05-25 11:34:49,831 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_201005251129_0001_m_000001_0 -> (127, 36) from hadoop-49.c.a.com
2010-05-25 11:34:50,797 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201005251129_0001_r_000006_0: Got 1 new map-outputs
2010-05-25 11:34:54,835 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201005251129_0001_r_000006_0 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2010-05-25 11:34:54,841 INFO org.apache.hadoop.mapred.ReduceTask: header: attempt_201005251129_0001_m_000000_0, compressed len: 1553902, decompressed len: 1553898
2010-05-25 11:34:54,841 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 1553898 bytes (1553902 raw bytes) into RAM from attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,924 INFO org.apache.hadoop.mapred.ReduceTask: Read 1553898 bytes from map-output for attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,944 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_201005251129_0001_m_000000_0 -> (143, 36) from hadoop-25.c.a.com
Reducer2:
2010-05-25 11:34:49,822 INFO org.apache.hadoop.mapred.ReduceTask: Read 637657 bytes from map-output for attempt_201005251129_0001_m_000001_0
2010-05-25 11:34:49,911 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_201005251129_0001_m_000001_0 -> (125, 36) from hadoop-49.c.a.com
2010-05-25 11:34:50,806 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201005251129_0001_r_000008_0: Got 1 new map-outputs
2010-05-25 11:34:54,915 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201005251129_0001_r_000008_0 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2010-05-25 11:34:54,920 INFO org.apache.hadoop.mapred.ReduceTask: header: attempt_201005251129_0001_m_000000_0, compressed len: 1462335, decompressed len: 1462331
2010-05-25 11:34:54,920 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 1462331 bytes (1462335 raw bytes) into RAM from attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,937 INFO org.apache.hadoop.mapred.ReduceTask: Read 1462331 bytes from map-output for attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,937 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_201005251129_0001_m_000000_0 -> (147, 36) from hadoop-25.c.a.com
The 2 reduce tasks have different task ids and belong to the same job.
Thanks,
Deepika
-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com]
Sent: Tuesday, May 25, 2010 8:10 AM
To: general@hadoop.apache.org
Subject: Re: Re: Hash Partitioner
On Mon, May 24, 2010 at 6:32 PM, Deepika Khera <De...@avg.com> wrote:
> Thanks for your response Eric.
>
> I am using hadoop 0.20.2.
>
> Here is what the hashCode() implementation looks like (I actually had the IDE generate it for me)
>
> Main key (for mapper & reducer):
>
> public int hashCode() {
> int result = kVersion;
> result = 31 * result + (aKey != null ? aKey.hashCode() : 0);
> result = 31 * result + (gKey != null ? gKey.hashCode() : 0);
> result = 31 * result + (int) (date ^ (date >>> 32));
> result = 31 * result + (ma != null ? ma.hashCode() : 0);
> result = 31 * result + (cl != null ? cl.hashCode() : 0);
> return result;
> }
>
>
> aKey : AKey class
>
>
> public int hashCode() {
> int result = kVersion;
> result = 31 * result + (v != null ? v.hashCode() : 0);
> result = 31 * result + (s != null ? s.hashCode() : 0);
> result = 31 * result + (o != null ? o.hashCode() : 0);
> result = 31 * result + (l != null ? l.hashCode() : 0);
> result = 31 * result + (e ? 1 : 0); //boolean
> result = 31 * result + (li ? 1 : 0); //boolean
> result = 31 * result + (aut ? 1 : 0); //boolean
> return result;
> }
>
Both of these look fine, assuming all the other hashCode()s return the
same value every time.
> When this happens, I do see the same values for the key. Also I am not using a grouping comparator.
So you see two reduce methods getting the same key with the same
values? That's extremely odd. If this is the case, there's a bug in
Hadoop. Can you find the relevant logs from the reducers where Hadoop
fetches the map output? Does it look like its fetching the same output
twice? Do the two tasks where you see the duplicates have the same
task ID? Can you confirm the reduce tasks are from the same job ID for
us?
> I was wondering since the call to HashPartitioner.getPartition() is done from a map task, several of which are running on different machines, is it possible that they get a different hashcode and hence get different reducers assigned even when the key is the same.
The hashCode() result should *always* be the same given the same
internal state. In other words, it should be consistent and stable. If
I have a string new String("hello world") it will always have the
exact same hashCode(). If this isn't true, you will get wildly
unpredictable results not just with Hadoop but with Java's
comparators, collections, etc.
--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com
RE: Re: Re: Hash Partitioner
Posted by Deepika Khera <De...@avg.com>.
So I ran my process again with some more logging and here is what I see.
I used my own HashPartitioner(basically copied hadoop's partitioner and added some logging for analysis).I printed here the Key and the reducer that is assigned to the key (based on the hash code).
My process triggered off 2 mappers (running on 2 different hadoop machines), hence both of these try to find reducers for the split file assigned to them. I see that for a same object key assigned to both these mappers, I am getting 2 different reducers allocated by the Partitioner.
In the reducers I see -
1) 2 different reducers (the ones that the partitioner assigned the key to) printing out the same Key (I did not print out the value as I thought that wouldn't matter)
2) Here are the logs from where reducers copies data from the mapper -
Reducer1:
2010-05-25 11:34:49,810 INFO org.apache.hadoop.mapred.ReduceTask: Read 1002612 bytes from map-output for attempt_201005251129_0001_m_000001_0
2010-05-25 11:34:49,831 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_201005251129_0001_m_000001_0 -> (127, 36) from hadoop-49.c.a.com
2010-05-25 11:34:50,797 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201005251129_0001_r_000006_0: Got 1 new map-outputs
2010-05-25 11:34:54,835 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201005251129_0001_r_000006_0 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2010-05-25 11:34:54,841 INFO org.apache.hadoop.mapred.ReduceTask: header: attempt_201005251129_0001_m_000000_0, compressed len: 1553902, decompressed len: 1553898
2010-05-25 11:34:54,841 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 1553898 bytes (1553902 raw bytes) into RAM from attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,924 INFO org.apache.hadoop.mapred.ReduceTask: Read 1553898 bytes from map-output for attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,944 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_201005251129_0001_m_000000_0 -> (143, 36) from hadoop-25.c.a.com
Reducer2:
2010-05-25 11:34:49,822 INFO org.apache.hadoop.mapred.ReduceTask: Read 637657 bytes from map-output for attempt_201005251129_0001_m_000001_0
2010-05-25 11:34:49,911 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_201005251129_0001_m_000001_0 -> (125, 36) from hadoop-49.c.a.com
2010-05-25 11:34:50,806 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201005251129_0001_r_000008_0: Got 1 new map-outputs
2010-05-25 11:34:54,915 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201005251129_0001_r_000008_0 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2010-05-25 11:34:54,920 INFO org.apache.hadoop.mapred.ReduceTask: header: attempt_201005251129_0001_m_000000_0, compressed len: 1462335, decompressed len: 1462331
2010-05-25 11:34:54,920 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 1462331 bytes (1462335 raw bytes) into RAM from attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,937 INFO org.apache.hadoop.mapred.ReduceTask: Read 1462331 bytes from map-output for attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,937 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 from attempt_201005251129_0001_m_000000_0 -> (147, 36) from hadoop-25.c.a.com
The 2 reduce tasks have different task ids and belong to the same job.
Thanks,
Deepika
-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com]
Sent: Tuesday, May 25, 2010 8:10 AM
To: general@hadoop.apache.org
Subject: Re: Re: Hash Partitioner
On Mon, May 24, 2010 at 6:32 PM, Deepika Khera <De...@avg.com> wrote:
> Thanks for your response Eric.
>
> I am using hadoop 0.20.2.
>
> Here is what the hashCode() implementation looks like (I actually had the IDE generate it for me)
>
> Main key (for mapper & reducer):
>
> public int hashCode() {
> int result = kVersion;
> result = 31 * result + (aKey != null ? aKey.hashCode() : 0);
> result = 31 * result + (gKey != null ? gKey.hashCode() : 0);
> result = 31 * result + (int) (date ^ (date >>> 32));
> result = 31 * result + (ma != null ? ma.hashCode() : 0);
> result = 31 * result + (cl != null ? cl.hashCode() : 0);
> return result;
> }
>
>
> aKey : AKey class
>
>
> public int hashCode() {
> int result = kVersion;
> result = 31 * result + (v != null ? v.hashCode() : 0);
> result = 31 * result + (s != null ? s.hashCode() : 0);
> result = 31 * result + (o != null ? o.hashCode() : 0);
> result = 31 * result + (l != null ? l.hashCode() : 0);
> result = 31 * result + (e ? 1 : 0); //boolean
> result = 31 * result + (li ? 1 : 0); //boolean
> result = 31 * result + (aut ? 1 : 0); //boolean
> return result;
> }
>
Both of these look fine, assuming all the other hashCode()s return the
same value every time.
> When this happens, I do see the same values for the key. Also I am not using a grouping comparator.
So you see two reduce methods getting the same key with the same
values? That's extremely odd. If this is the case, there's a bug in
Hadoop. Can you find the relevant logs from the reducers where Hadoop
fetches the map output? Does it look like its fetching the same output
twice? Do the two tasks where you see the duplicates have the same
task ID? Can you confirm the reduce tasks are from the same job ID for
us?
> I was wondering since the call to HashPartitioner.getPartition() is done from a map task, several of which are running on different machines, is it possible that they get a different hashcode and hence get different reducers assigned even when the key is the same.
The hashCode() result should *always* be the same given the same
internal state. In other words, it should be consistent and stable. If
I have a string new String("hello world") it will always have the
exact same hashCode(). If this isn't true, you will get wildly
unpredictable results not just with Hadoop but with Java's
comparators, collections, etc.
--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com
Re: Hash Partitioner
Posted by "Ankur C. Goel" <ga...@yahoo-inc.com>.
Another thing I would look at critically is to see if there are any bugs in the write() and readFields() method of my key class as there could hard to identify serialization / de-serialization issues.
-@nkur
On 5/25/10 8:39 PM, "Eric Sammer" <es...@cloudera.com> wrote:
On Mon, May 24, 2010 at 6:32 PM, Deepika Khera <De...@avg.com> wrote:
> Thanks for your response Eric.
>
> I am using hadoop 0.20.2.
>
> Here is what the hashCode() implementation looks like (I actually had the IDE generate it for me)
>
> Main key (for mapper & reducer):
>
> public int hashCode() {
> int result = kVersion;
> result = 31 * result + (aKey != null ? aKey.hashCode() : 0);
> result = 31 * result + (gKey != null ? gKey.hashCode() : 0);
> result = 31 * result + (int) (date ^ (date >>> 32));
> result = 31 * result + (ma != null ? ma.hashCode() : 0);
> result = 31 * result + (cl != null ? cl.hashCode() : 0);
> return result;
> }
>
>
> aKey : AKey class
>
>
> public int hashCode() {
> int result = kVersion;
> result = 31 * result + (v != null ? v.hashCode() : 0);
> result = 31 * result + (s != null ? s.hashCode() : 0);
> result = 31 * result + (o != null ? o.hashCode() : 0);
> result = 31 * result + (l != null ? l.hashCode() : 0);
> result = 31 * result + (e ? 1 : 0); //boolean
> result = 31 * result + (li ? 1 : 0); //boolean
> result = 31 * result + (aut ? 1 : 0); //boolean
> return result;
> }
>
Both of these look fine, assuming all the other hashCode()s return the
same value every time.
> When this happens, I do see the same values for the key. Also I am not using a grouping comparator.
So you see two reduce methods getting the same key with the same
values? That's extremely odd. If this is the case, there's a bug in
Hadoop. Can you find the relevant logs from the reducers where Hadoop
fetches the map output? Does it look like its fetching the same output
twice? Do the two tasks where you see the duplicates have the same
task ID? Can you confirm the reduce tasks are from the same job ID for
us?
> I was wondering since the call to HashPartitioner.getPartition() is done from a map task, several of which are running on different machines, is it possible that they get a different hashcode and hence get different reducers assigned even when the key is the same.
The hashCode() result should *always* be the same given the same
internal state. In other words, it should be consistent and stable. If
I have a string new String("hello world") it will always have the
exact same hashCode(). If this isn't true, you will get wildly
unpredictable results not just with Hadoop but with Java's
comparators, collections, etc.
--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com
Re: Re: Hash Partitioner
Posted by Eric Sammer <es...@cloudera.com>.
On Mon, May 24, 2010 at 6:32 PM, Deepika Khera <De...@avg.com> wrote:
> Thanks for your response Eric.
>
> I am using hadoop 0.20.2.
>
> Here is what the hashCode() implementation looks like (I actually had the IDE generate it for me)
>
> Main key (for mapper & reducer):
>
> public int hashCode() {
> int result = kVersion;
> result = 31 * result + (aKey != null ? aKey.hashCode() : 0);
> result = 31 * result + (gKey != null ? gKey.hashCode() : 0);
> result = 31 * result + (int) (date ^ (date >>> 32));
> result = 31 * result + (ma != null ? ma.hashCode() : 0);
> result = 31 * result + (cl != null ? cl.hashCode() : 0);
> return result;
> }
>
>
> aKey : AKey class
>
>
> public int hashCode() {
> int result = kVersion;
> result = 31 * result + (v != null ? v.hashCode() : 0);
> result = 31 * result + (s != null ? s.hashCode() : 0);
> result = 31 * result + (o != null ? o.hashCode() : 0);
> result = 31 * result + (l != null ? l.hashCode() : 0);
> result = 31 * result + (e ? 1 : 0); //boolean
> result = 31 * result + (li ? 1 : 0); //boolean
> result = 31 * result + (aut ? 1 : 0); //boolean
> return result;
> }
>
Both of these look fine, assuming all the other hashCode()s return the
same value every time.
> When this happens, I do see the same values for the key. Also I am not using a grouping comparator.
So you see two reduce methods getting the same key with the same
values? That's extremely odd. If this is the case, there's a bug in
Hadoop. Can you find the relevant logs from the reducers where Hadoop
fetches the map output? Does it look like its fetching the same output
twice? Do the two tasks where you see the duplicates have the same
task ID? Can you confirm the reduce tasks are from the same job ID for
us?
> I was wondering since the call to HashPartitioner.getPartition() is done from a map task, several of which are running on different machines, is it possible that they get a different hashcode and hence get different reducers assigned even when the key is the same.
The hashCode() result should *always* be the same given the same
internal state. In other words, it should be consistent and stable. If
I have a string new String("hello world") it will always have the
exact same hashCode(). If this isn't true, you will get wildly
unpredictable results not just with Hadoop but with Java's
comparators, collections, etc.
--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com
RE: Re: Hash Partitioner
Posted by Deepika Khera <De...@avg.com>.
Thanks for your response Eric.
I am using hadoop 0.20.2.
Here is what the hashCode() implementation looks like (I actually had the IDE generate it for me)
Main key (for mapper & reducer):
public int hashCode() {
int result = kVersion;
result = 31 * result + (aKey != null ? aKey.hashCode() : 0);
result = 31 * result + (gKey != null ? gKey.hashCode() : 0);
result = 31 * result + (int) (date ^ (date >>> 32));
result = 31 * result + (ma != null ? ma.hashCode() : 0);
result = 31 * result + (cl != null ? cl.hashCode() : 0);
return result;
}
aKey : AKey class
public int hashCode() {
int result = kVersion;
result = 31 * result + (v != null ? v.hashCode() : 0);
result = 31 * result + (s != null ? s.hashCode() : 0);
result = 31 * result + (o != null ? o.hashCode() : 0);
result = 31 * result + (l != null ? l.hashCode() : 0);
result = 31 * result + (e ? 1 : 0); //boolean
result = 31 * result + (li ? 1 : 0); //boolean
result = 31 * result + (aut ? 1 : 0); //boolean
return result;
}
When this happens, I do see the same values for the key. Also I am not using a grouping comparator.
I was wondering since the call to HashPartitioner.getPartition() is done from a map task, several of which are running on different machines, is it possible that they get a different hashcode and hence get different reducers assigned even when the key is the same.
Thanks,
Deepika
-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com]
Sent: Monday, May 24, 2010 3:07 PM
To: general@hadoop.apache.org
Subject: Re: Hash Partitioner
Deepika:
That sounds very strange. Can you let us know what version of Hadoop
(e.g. Apache 0.20.x, CDH2, etc.) you're running and a bit more about
your hashCode() implementation? When this happens, do you see the same
values for the duplicate key? Did you also implement a grouping
comparator?
The hash partitioner is extremely simple. It basically does
key.hashCode() % numberOfReduces = partition number to which a key is
assigned. If one incorrectly implements a grouping comparator, it's
possible you could see odd behavior, though.
On Mon, May 24, 2010 at 5:35 PM, Deepika Khera <De...@avg.com> wrote:
> Hi,
>
> I am using a HashPartitioner on my key for a map reducer job. I am wondering how sometimes 2 reducers end up getting the same key ? I have the hashCode method defined for my key.
>
> Also, I have speculative execution turned off for my jobs..
>
> Would appreciate any help.
>
> Thanks,
> Deepika
>
--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com
Re: Hash Partitioner
Posted by Eric Sammer <es...@cloudera.com>.
Deepika:
That sounds very strange. Can you let us know what version of Hadoop
(e.g. Apache 0.20.x, CDH2, etc.) you're running and a bit more about
your hashCode() implementation? When this happens, do you see the same
values for the duplicate key? Did you also implement a grouping
comparator?
The hash partitioner is extremely simple. It basically does
key.hashCode() % numberOfReduces = partition number to which a key is
assigned. If one incorrectly implements a grouping comparator, it's
possible you could see odd behavior, though.
On Mon, May 24, 2010 at 5:35 PM, Deepika Khera <De...@avg.com> wrote:
> Hi,
>
> I am using a HashPartitioner on my key for a map reducer job. I am wondering how sometimes 2 reducers end up getting the same key ? I have the hashCode method defined for my key.
>
> Also, I have speculative execution turned off for my jobs..
>
> Would appreciate any help.
>
> Thanks,
> Deepika
>
--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com