You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Grant Ingersoll <gs...@apache.org> on 2008/10/28 03:05:12 UTC
"Merge of the inmemory files threw an exception" and diffs between 0.17.2 and 0.18.1
Hi,
Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with some
of our clustering code and Hadoop 0.18.1. The thread in context is
at: http://mahout.markmail.org/message/vcyvlz2met7fnthr
The problem seems to occur when going from 0.17.2 to 0.18.1. In the
user logs, we are seeing the following exception:
2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 2 segments left of total size: 5011 bytes
2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200810272112_0011_r_000000_0 Merge of the inmemory files threw
an exception: java.io.IOException: Intermedate merge failed
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.run(ReduceTask.java:2078)
Caused by: java.lang.NumberFormatException: For input string: "["
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1224)
at java.lang.Double.parseDouble(Double.java:510)
at
org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
at
org
.apache.mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:
256)
at
org
.apache
.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:38)
at
org
.apache
.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:31)
at org.apache.hadoop.mapred.ReduceTask
$ReduceCopier.combineAndSpill(ReduceTask.java:2174)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
$3100(ReduceTask.java:341)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
And in the main output log (from running bin/hadoop jar mahout/
examples/build/apache-mahout-examples-0.1-dev.job
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
attempt_200810272112_0011_r_000000_0, Status : FAILED
java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
at org.apache.hadoop.mapred.TaskTracker
$Child.main(TaskTracker.java:2207)
If I run this exact same job on 0.17.2 it all runs fine. I suppose
either a bug was introduced in 0.18.1 or a bug was fixed that we were
relying on. Looking at the release notes between the fixes, nothing
in particular struck me as related. If it helps, I can provide the
instructions for how to run the example in question (they need to be
written up anyway!)
I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
, but those are older, it seems, so not sure what to make of them.
Thanks,
Grant
RE: "Merge of the inmemory files threw an exception" and diffs between 0.17.2 and 0.18.1
Posted by Deepika Khera <de...@collarity.com>.
Wow, if the issue is fixed with version 0.20, then could we please have
a patch for version 0.18?
Thanks,
Deepika
-----Original Message-----
From: Grant Ingersoll [mailto:gsingers@apache.org]
Sent: Thursday, October 30, 2008 12:19 PM
To: core-user@hadoop.apache.org
Subject: Re: "Merge of the inmemory files threw an exception" and diffs
between 0.17.2 and 0.18.1
So, Philippe reports that the problem goes away with 0.20-dev
(trunk?): http://mahout.markmail.org/message/swmzreg6fnzf6icv We
aren't totally clear on the structure of SVN for Hadoop, but it seems
like it is not fixed by this patch.
On Oct 29, 2008, at 10:28 AM, Grant Ingersoll wrote:
> We'll try it out...
>
> On Oct 28, 2008, at 3:00 PM, Arun C Murthy wrote:
>
>>
>> On Oct 27, 2008, at 7:05 PM, Grant Ingersoll wrote:
>>
>>> Hi,
>>>
>>> Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
>>> some of our clustering code and Hadoop 0.18.1. The thread in
>>> context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr
>>>
>>> The problem seems to occur when going from 0.17.2 to 0.18.1. In
>>> the user logs, we are seeing the following exception:
>>> 2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
>>> to the last merge-pass, with 2 segments left of total size: 5011
>>> bytes
>>> 2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
>>> attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
>>> threw an exception: java.io.IOException: Intermedate merge failed
>>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
>>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>>> $InMemFSMergeThread.run(ReduceTask.java:2078)
>>> Caused by: java.lang.NumberFormatException: For input string: "["
>>
>> If you are sure that this isn't caused by your application-logic,
>> you could try running with
http://issues.apache.org/jira/browse/HADOOP-4277
>> .
>>
>> That bug caused many a ship to sail in large circles, hopelessly.
>>
>> Arun
>>
>>>
>>> at
>>> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
>>> 1224)
>>> at java.lang.Double.parseDouble(Double.java:510)
>>> at
>>> org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:
>>> 60)
>>> at
>>> org
>>> .apache
>>> .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
>>> at
>>> org
>>> .apache
>>> .mahout
>>> .clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:38)
>>> at
>>> org
>>> .apache
>>> .mahout
>>> .clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:31)
>>> at org.apache.hadoop.mapred.ReduceTask
>>> $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
>>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
>>> $3100(ReduceTask.java:341)
>>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
>>>
>>> And in the main output log (from running bin/hadoop jar mahout/
>>> examples/build/apache-mahout-examples-0.1-dev.job
>>> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
>>> 08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
>>> attempt_200810272112_0011_r_000000_0, Status : FAILED
>>> java.io.IOException: attempt_200810272112_0011_r_000000_0The
>>> reduce copier failed
>>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
>>> at org.apache.hadoop.mapred.TaskTracker
>>> $Child.main(TaskTracker.java:2207)
>>>
>>> If I run this exact same job on 0.17.2 it all runs fine. I
>>> suppose either a bug was introduced in 0.18.1 or a bug was fixed
>>> that we were relying on. Looking at the release notes between the
>>> fixes, nothing in particular struck me as related. If it helps, I
>>> can provide the instructions for how to run the example in
>>> question (they need to be written up anyway!)
>>>
>>>
>>> I see some related things at
http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
n+exception
>>> , but those are older, it seems, so not sure what to make of them.
>>>
>>> Thanks,
>>> Grant
>>
>
> --------------------------
> Grant Ingersoll
> Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
> http://www.lucenebootcamp.com
>
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
>
Re: "Merge of the inmemory files threw an exception" and diffs between 0.17.2 and 0.18.1
Posted by Grant Ingersoll <gs...@apache.org>.
So, Philippe reports that the problem goes away with 0.20-dev
(trunk?): http://mahout.markmail.org/message/swmzreg6fnzf6icv We
aren't totally clear on the structure of SVN for Hadoop, but it seems
like it is not fixed by this patch.
On Oct 29, 2008, at 10:28 AM, Grant Ingersoll wrote:
> We'll try it out...
>
> On Oct 28, 2008, at 3:00 PM, Arun C Murthy wrote:
>
>>
>> On Oct 27, 2008, at 7:05 PM, Grant Ingersoll wrote:
>>
>>> Hi,
>>>
>>> Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
>>> some of our clustering code and Hadoop 0.18.1. The thread in
>>> context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr
>>>
>>> The problem seems to occur when going from 0.17.2 to 0.18.1. In
>>> the user logs, we are seeing the following exception:
>>> 2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
>>> to the last merge-pass, with 2 segments left of total size: 5011
>>> bytes
>>> 2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
>>> attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
>>> threw an exception: java.io.IOException: Intermedate merge failed
>>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
>>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>>> $InMemFSMergeThread.run(ReduceTask.java:2078)
>>> Caused by: java.lang.NumberFormatException: For input string: "["
>>
>> If you are sure that this isn't caused by your application-logic,
>> you could try running with http://issues.apache.org/jira/browse/HADOOP-4277
>> .
>>
>> That bug caused many a ship to sail in large circles, hopelessly.
>>
>> Arun
>>
>>>
>>> at
>>> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
>>> 1224)
>>> at java.lang.Double.parseDouble(Double.java:510)
>>> at
>>> org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:
>>> 60)
>>> at
>>> org
>>> .apache
>>> .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
>>> at
>>> org
>>> .apache
>>> .mahout
>>> .clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:38)
>>> at
>>> org
>>> .apache
>>> .mahout
>>> .clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:31)
>>> at org.apache.hadoop.mapred.ReduceTask
>>> $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
>>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
>>> $3100(ReduceTask.java:341)
>>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
>>>
>>> And in the main output log (from running bin/hadoop jar mahout/
>>> examples/build/apache-mahout-examples-0.1-dev.job
>>> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
>>> 08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
>>> attempt_200810272112_0011_r_000000_0, Status : FAILED
>>> java.io.IOException: attempt_200810272112_0011_r_000000_0The
>>> reduce copier failed
>>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
>>> at org.apache.hadoop.mapred.TaskTracker
>>> $Child.main(TaskTracker.java:2207)
>>>
>>> If I run this exact same job on 0.17.2 it all runs fine. I
>>> suppose either a bug was introduced in 0.18.1 or a bug was fixed
>>> that we were relying on. Looking at the release notes between the
>>> fixes, nothing in particular struck me as related. If it helps, I
>>> can provide the instructions for how to run the example in
>>> question (they need to be written up anyway!)
>>>
>>>
>>> I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
>>> , but those are older, it seems, so not sure what to make of them.
>>>
>>> Thanks,
>>> Grant
>>
>
> --------------------------
> Grant Ingersoll
> Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
> http://www.lucenebootcamp.com
>
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
>
Re: "Merge of the inmemory files threw an exception" and diffs between 0.17.2 and 0.18.1
Posted by Grant Ingersoll <gs...@apache.org>.
We'll try it out...
On Oct 28, 2008, at 3:00 PM, Arun C Murthy wrote:
>
> On Oct 27, 2008, at 7:05 PM, Grant Ingersoll wrote:
>
>> Hi,
>>
>> Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
>> some of our clustering code and Hadoop 0.18.1. The thread in
>> context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr
>>
>> The problem seems to occur when going from 0.17.2 to 0.18.1. In
>> the user logs, we are seeing the following exception:
>> 2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
>> to the last merge-pass, with 2 segments left of total size: 5011
>> bytes
>> 2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
>> threw an exception: java.io.IOException: Intermedate merge failed
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.run(ReduceTask.java:2078)
>> Caused by: java.lang.NumberFormatException: For input string: "["
>
> If you are sure that this isn't caused by your application-logic,
> you could try running with http://issues.apache.org/jira/browse/HADOOP-4277
> .
>
> That bug caused many a ship to sail in large circles, hopelessly.
>
> Arun
>
>>
>> at
>> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
>> 1224)
>> at java.lang.Double.parseDouble(Double.java:510)
>> at
>> org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:
>> 60)
>> at
>> org
>> .apache
>> .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
>> at
>> org
>> .apache
>> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
>> 38)
>> at
>> org
>> .apache
>> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
>> 31)
>> at org.apache.hadoop.mapred.ReduceTask
>> $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
>> $3100(ReduceTask.java:341)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
>>
>> And in the main output log (from running bin/hadoop jar mahout/
>> examples/build/apache-mahout-examples-0.1-dev.job
>> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
>> 08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
>> attempt_200810272112_0011_r_000000_0, Status : FAILED
>> java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
>> copier failed
>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
>> at org.apache.hadoop.mapred.TaskTracker
>> $Child.main(TaskTracker.java:2207)
>>
>> If I run this exact same job on 0.17.2 it all runs fine. I suppose
>> either a bug was introduced in 0.18.1 or a bug was fixed that we
>> were relying on. Looking at the release notes between the fixes,
>> nothing in particular struck me as related. If it helps, I can
>> provide the instructions for how to run the example in question
>> (they need to be written up anyway!)
>>
>>
>> I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
>> , but those are older, it seems, so not sure what to make of them.
>>
>> Thanks,
>> Grant
>
--------------------------
Grant Ingersoll
Lucene Boot Camp Training Nov. 3-4, 2008, ApacheCon US New Orleans.
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
Re: "Merge of the inmemory files threw an exception" and diffs between 0.17.2 and 0.18.1
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Oct 27, 2008, at 7:05 PM, Grant Ingersoll wrote:
> Hi,
>
> Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
> some of our clustering code and Hadoop 0.18.1. The thread in
> context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr
>
> The problem seems to occur when going from 0.17.2 to 0.18.1. In the
> user logs, we are seeing the following exception:
> 2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
> to the last merge-pass, with 2 segments left of total size: 5011 bytes
> 2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
> threw an exception: java.io.IOException: Intermedate merge failed
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $InMemFSMergeThread.run(ReduceTask.java:2078)
> Caused by: java.lang.NumberFormatException: For input string: "["
If you are sure that this isn't caused by your application-logic, you
could try running with http://issues.apache.org/jira/browse/HADOOP-4277.
That bug caused many a ship to sail in large circles, hopelessly.
Arun
>
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
> 1224)
> at java.lang.Double.parseDouble(Double.java:510)
> at
> org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
> at
> org
> .apache
> .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
> at
> org
> .apache
> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
> 38)
> at
> org
> .apache
> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
> 31)
> at org.apache.hadoop.mapred.ReduceTask
> $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
> $3100(ReduceTask.java:341)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
>
> And in the main output log (from running bin/hadoop jar mahout/
> examples/build/apache-mahout-examples-0.1-dev.job
> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
> 08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
> attempt_200810272112_0011_r_000000_0, Status : FAILED
> java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
> copier failed
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
> at org.apache.hadoop.mapred.TaskTracker
> $Child.main(TaskTracker.java:2207)
>
> If I run this exact same job on 0.17.2 it all runs fine. I suppose
> either a bug was introduced in 0.18.1 or a bug was fixed that we
> were relying on. Looking at the release notes between the fixes,
> nothing in particular struck me as related. If it helps, I can
> provide the instructions for how to run the example in question
> (they need to be written up anyway!)
>
>
> I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
> , but those are older, it seems, so not sure what to make of them.
>
> Thanks,
> Grant
RE: "Merge of the inmemory files threw an exception" and diffs between 0.17.2 and 0.18.1
Posted by Deepika Khera <de...@collarity.com>.
Hi Devraj,
It was pretty consistent with my comparator class in my old email(the
one that uses UTF8). While trying to resolve the issue, I changed UTF8
to Text. That made it disappear for a while but then it came back again.
My new Comparator class(with Text) is -
public class IncrementalURLIndexKey implements WritableComparable {
private Text url;
private long userid;
public IncrementalURLIndexKey() {
}
public IncrementalURLIndexKey(Text url, long userid) {
this.url = url;
this.userid = userid;
}
public Text getUrl() {
return url;
}
public long getUserid() {
return userid;
}
public void write(DataOutput out) throws IOException {
url.write(out);
out.writeLong(userid);
}
public void readFields(DataInput in) throws IOException {
url = new Text();
url.readFields(in);
userid = in.readLong();
}
public int compareTo(Object o) {
IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
int result = url.compareTo(other.getUrl());
if (result == 0) result = CUID.compare(userid, other.userid);
return result;
}
/**
* A Comparator optimized for IncrementalURLIndexKey.
*/
public static class GroupingComparator extends WritableComparator {
public GroupingComparator() {
super(IncrementalURLIndexKey.class, true);
}
public int compare(WritableComparable a, WritableComparable b) {
IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;
return key1.getUrl().compareTo(key2.getUrl());
}
}
static {
WritableComparator.define(IncrementalURLIndexKey.class, new
GroupingComparator());
}
}
Thanks,
Deepika
-----Original Message-----
From: Devaraj Das [mailto:ddas@yahoo-inc.com]
Sent: Tuesday, October 28, 2008 9:01 PM
To: core-user@hadoop.apache.org
Subject: Re: "Merge of the inmemory files threw an exception" and diffs
between 0.17.2 and 0.18.1
Quick question (I haven't looked at your comparator code yet) - is this
reproducible/consistent?
On 10/28/08 11:52 PM, "Deepika Khera" <de...@collarity.com> wrote:
> I am getting a similar exception too with Hadoop 0.18.1(See stacktrace
> below), though its an EOFException. Does anyone have any idea about
what
> it means and how it can be fixed?
>
> 2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200810241922_0844_r_000006_0 Merge of the inmemory files threw
> an exception: java.io.IOException: Intermedate merge failed
> at
>
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
> MemMerge(ReduceTask.java:2147)
> at
>
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(
> ReduceTask.java:2078)
> Caused by: java.lang.RuntimeException: java.io.EOFException
> at
>
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 103)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
> at
> org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:122)
> at
> org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:49)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:321)
> at org.apache.hadoop.mapred.Merger.merge(Merger.java:72)
> at
>
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
> MemMerge(ReduceTask.java:2123)
> ... 1 more
> Caused by: java.io.EOFException
> at
> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:103)
> at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
> at
>
com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
> ey.java:40)
> at
>
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 97)
> ... 7 more
>
> 2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200810241922_0844_r_000006_0 Merging of the local FS files
threw
> an exception: java.io.IOException: java.lang.RuntimeException:
> java.io.EOFException
> at
>
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 103)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
> at
> org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:135)
> at
> org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:102)
> at
>
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.ja
> va:226)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242)
> at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
> at
>
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
> eTask.java:2035)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
> at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
> at
>
com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
> ey.java:40)
> at
>
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 97)
> ... 7 more
>
> at
>
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
> eTask.java:2039)
>
> 2008-10-27 16:53:07,907 WARN org.apache.hadoop.mapred.TaskTracker:
Error
> running child
> java.io.IOException: attempt_200810241922_0844_r_000006_0The reduce
> copier failed
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
>
>
>
>
> My WritableComparable class looks like this -
>
>
>
>
> public class IncrementalURLIndexKey implements WritableComparable {
> private UTF8 url;
> private long userid;
>
> public IncrementalURLIndexKey() {
> }
>
> public IncrementalURLIndexKey(UTF8 url, long userid) {
> this.url = url;
> this.userid = userid;
> }
>
> public UTF8 getUrl() {
> return url;
> }
>
> public long getUserid() {
> return userid;
> }
>
> public void write(DataOutput out) throws IOException {
> IOUtil.writeUTF8(out, url);
> out.writeLong(userid);
> }
>
> public void readFields(DataInput in) throws IOException {
> url = IOUtil.readUTF8(in, url);
> userid = in.readLong();
> }
>
> public int compareTo(Object o) {
> IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
> int result = url.compareTo(other.getUrl());
> if (result == 0) result = CUID.compare(userid, other.userid);
> return result;
> }
>
> /**
> * A Comparator optimized for IncrementalURLIndexKey.
> */
> public static class GroupingComparator extends WritableComparator {
> public GroupingComparator() {
> super(IncrementalURLIndexKey.class, true);
> }
>
> public int compare(WritableComparable a, WritableComparable b) {
> IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
> IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;
>
> return key1.getUrl().compareTo(key2.getUrl());
> }
> }
>
> static {
> WritableComparator.define(IncrementalURLIndexKey.class, new
> GroupingComparator());
> }
> }
>
>
> Thanks,
> Deepika
>
> -----Original Message-----
> From: Grant Ingersoll [mailto:gsingers@apache.org]
> Sent: Monday, October 27, 2008 7:14 PM
> To: core-user@hadoop.apache.org
> Subject: Re: "Merge of the inmemory files threw an exception" and
diffs
> between 0.17.2 and 0.18.1
>
> I suppose I should add an actual question: Is this a bug or are we
> relying on a "feature" that has since been fixed?
>
> On Oct 27, 2008, at 10:05 PM, Grant Ingersoll wrote:
>
>> Hi,
>>
>> Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
>> some of our clustering code and Hadoop 0.18.1. The thread in
>> context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr
>>
>> The problem seems to occur when going from 0.17.2 to 0.18.1. In the
>> user logs, we are seeing the following exception:
>> 2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
>> to the last merge-pass, with 2 segments left of total size: 5011
bytes
>> 2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
>> threw an exception: java.io.IOException: Intermedate merge failed
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.run(ReduceTask.java:2078)
>> Caused by: java.lang.NumberFormatException: For input string: "["
>> at
>> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
>> 1224)
>> at java.lang.Double.parseDouble(Double.java:510)
>> at
>>
org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
>> at
>> org
>> .apache
>> .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
>> at
>> org
>> .apache
>> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
>> 38)
>> at
>> org
>> .apache
>> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
>> 31)
>> at org.apache.hadoop.mapred.ReduceTask
>> $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
>> $3100(ReduceTask.java:341)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
>>
>> And in the main output log (from running bin/hadoop jar mahout/
>> examples/build/apache-mahout-examples-0.1-dev.job
>> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
>> 08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
>> attempt_200810272112_0011_r_000000_0, Status : FAILED
>> java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
>> copier failed
>> at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
>> at org.apache.hadoop.mapred.TaskTracker
>> $Child.main(TaskTracker.java:2207)
>>
>> If I run this exact same job on 0.17.2 it all runs fine. I suppose
>> either a bug was introduced in 0.18.1 or a bug was fixed that we
>> were relying on. Looking at the release notes between the fixes,
>> nothing in particular struck me as related. If it helps, I can
>> provide the instructions for how to run the example in question
>> (they need to be written up anyway!)
>>
>>
>> I see some related things at
>
http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
> n+exception
>> , but those are older, it seems, so not sure what to make of them.
>>
>> Thanks,
>> Grant
>
>
Re: "Merge of the inmemory files threw an exception" and diffs
between 0.17.2 and 0.18.1
Posted by Devaraj Das <dd...@yahoo-inc.com>.
Quick question (I haven't looked at your comparator code yet) - is this
reproducible/consistent?
On 10/28/08 11:52 PM, "Deepika Khera" <de...@collarity.com> wrote:
> I am getting a similar exception too with Hadoop 0.18.1(See stacktrace
> below), though its an EOFException. Does anyone have any idea about what
> it means and how it can be fixed?
>
> 2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200810241922_0844_r_000006_0 Merge of the inmemory files threw
> an exception: java.io.IOException: Intermedate merge failed
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
> MemMerge(ReduceTask.java:2147)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(
> ReduceTask.java:2078)
> Caused by: java.lang.RuntimeException: java.io.EOFException
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 103)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
> at
> org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:122)
> at
> org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:49)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:321)
> at org.apache.hadoop.mapred.Merger.merge(Merger.java:72)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
> MemMerge(ReduceTask.java:2123)
> ... 1 more
> Caused by: java.io.EOFException
> at
> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:103)
> at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
> at
> com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
> ey.java:40)
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 97)
> ... 7 more
>
> 2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200810241922_0844_r_000006_0 Merging of the local FS files threw
> an exception: java.io.IOException: java.lang.RuntimeException:
> java.io.EOFException
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 103)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
> at
> org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:135)
> at
> org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:102)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.ja
> va:226)
> at
> org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242)
> at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
> eTask.java:2035)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
> at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
> at
> com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
> ey.java:40)
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
> 97)
> ... 7 more
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
> eTask.java:2039)
>
> 2008-10-27 16:53:07,907 WARN org.apache.hadoop.mapred.TaskTracker: Error
> running child
> java.io.IOException: attempt_200810241922_0844_r_000006_0The reduce
> copier failed
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
>
>
>
>
> My WritableComparable class looks like this -
>
>
>
>
> public class IncrementalURLIndexKey implements WritableComparable {
> private UTF8 url;
> private long userid;
>
> public IncrementalURLIndexKey() {
> }
>
> public IncrementalURLIndexKey(UTF8 url, long userid) {
> this.url = url;
> this.userid = userid;
> }
>
> public UTF8 getUrl() {
> return url;
> }
>
> public long getUserid() {
> return userid;
> }
>
> public void write(DataOutput out) throws IOException {
> IOUtil.writeUTF8(out, url);
> out.writeLong(userid);
> }
>
> public void readFields(DataInput in) throws IOException {
> url = IOUtil.readUTF8(in, url);
> userid = in.readLong();
> }
>
> public int compareTo(Object o) {
> IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
> int result = url.compareTo(other.getUrl());
> if (result == 0) result = CUID.compare(userid, other.userid);
> return result;
> }
>
> /**
> * A Comparator optimized for IncrementalURLIndexKey.
> */
> public static class GroupingComparator extends WritableComparator {
> public GroupingComparator() {
> super(IncrementalURLIndexKey.class, true);
> }
>
> public int compare(WritableComparable a, WritableComparable b) {
> IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
> IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;
>
> return key1.getUrl().compareTo(key2.getUrl());
> }
> }
>
> static {
> WritableComparator.define(IncrementalURLIndexKey.class, new
> GroupingComparator());
> }
> }
>
>
> Thanks,
> Deepika
>
> -----Original Message-----
> From: Grant Ingersoll [mailto:gsingers@apache.org]
> Sent: Monday, October 27, 2008 7:14 PM
> To: core-user@hadoop.apache.org
> Subject: Re: "Merge of the inmemory files threw an exception" and diffs
> between 0.17.2 and 0.18.1
>
> I suppose I should add an actual question: Is this a bug or are we
> relying on a "feature" that has since been fixed?
>
> On Oct 27, 2008, at 10:05 PM, Grant Ingersoll wrote:
>
>> Hi,
>>
>> Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
>> some of our clustering code and Hadoop 0.18.1. The thread in
>> context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr
>>
>> The problem seems to occur when going from 0.17.2 to 0.18.1. In the
>> user logs, we are seeing the following exception:
>> 2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
>> to the last merge-pass, with 2 segments left of total size: 5011 bytes
>> 2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
>> attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
>> threw an exception: java.io.IOException: Intermedate merge failed
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.run(ReduceTask.java:2078)
>> Caused by: java.lang.NumberFormatException: For input string: "["
>> at
>> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
>> 1224)
>> at java.lang.Double.parseDouble(Double.java:510)
>> at
>> org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
>> at
>> org
>> .apache
>> .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
>> at
>> org
>> .apache
>> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
>> 38)
>> at
>> org
>> .apache
>> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
>> 31)
>> at org.apache.hadoop.mapred.ReduceTask
>> $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
>> $3100(ReduceTask.java:341)
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
>> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
>>
>> And in the main output log (from running bin/hadoop jar mahout/
>> examples/build/apache-mahout-examples-0.1-dev.job
>> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
>> 08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
>> attempt_200810272112_0011_r_000000_0, Status : FAILED
>> java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
>> copier failed
>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
>> at org.apache.hadoop.mapred.TaskTracker
>> $Child.main(TaskTracker.java:2207)
>>
>> If I run this exact same job on 0.17.2 it all runs fine. I suppose
>> either a bug was introduced in 0.18.1 or a bug was fixed that we
>> were relying on. Looking at the release notes between the fixes,
>> nothing in particular struck me as related. If it helps, I can
>> provide the instructions for how to run the example in question
>> (they need to be written up anyway!)
>>
>>
>> I see some related things at
> http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
> n+exception
>> , but those are older, it seems, so not sure what to make of them.
>>
>> Thanks,
>> Grant
>
>
RE: "Merge of the inmemory files threw an exception" and diffs between 0.17.2 and 0.18.1
Posted by Deepika Khera <de...@collarity.com>.
I am getting a similar exception too with Hadoop 0.18.1(See stacktrace
below), though its an EOFException. Does anyone have any idea about what
it means and how it can be fixed?
2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200810241922_0844_r_000006_0 Merge of the inmemory files threw
an exception: java.io.IOException: Intermedate merge failed
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
MemMerge(ReduceTask.java:2147)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(
ReduceTask.java:2078)
Caused by: java.lang.RuntimeException: java.io.EOFException
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
103)
at
org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
at
org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:122)
at
org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:49)
at
org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:321)
at org.apache.hadoop.mapred.Merger.merge(Merger.java:72)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doIn
MemMerge(ReduceTask.java:2123)
... 1 more
Caused by: java.io.EOFException
at
java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
at org.apache.hadoop.io.UTF8.readFields(UTF8.java:103)
at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
at
com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
ey.java:40)
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
97)
... 7 more
2008-10-27 16:53:07,407 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200810241922_0844_r_000006_0 Merging of the local FS files threw
an exception: java.io.IOException: java.lang.RuntimeException:
java.io.EOFException
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
103)
at
org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:269)
at
org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:135)
at
org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:102)
at
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.ja
va:226)
at
org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:242)
at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
eTask.java:2035)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at org.apache.hadoop.io.UTF8.readFields(UTF8.java:106)
at com.collarity.io.IOUtil.readUTF8(IOUtil.java:213)
at
com.collarity.url.IncrementalURLIndexKey.readFields(IncrementalURLIndexK
ey.java:40)
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:
97)
... 7 more
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(Reduc
eTask.java:2039)
2008-10-27 16:53:07,907 WARN org.apache.hadoop.mapred.TaskTracker: Error
running child
java.io.IOException: attempt_200810241922_0844_r_000006_0The reduce
copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
My WritableComparable class looks like this -
public class IncrementalURLIndexKey implements WritableComparable {
private UTF8 url;
private long userid;
public IncrementalURLIndexKey() {
}
public IncrementalURLIndexKey(UTF8 url, long userid) {
this.url = url;
this.userid = userid;
}
public UTF8 getUrl() {
return url;
}
public long getUserid() {
return userid;
}
public void write(DataOutput out) throws IOException {
IOUtil.writeUTF8(out, url);
out.writeLong(userid);
}
public void readFields(DataInput in) throws IOException {
url = IOUtil.readUTF8(in, url);
userid = in.readLong();
}
public int compareTo(Object o) {
IncrementalURLIndexKey other = (IncrementalURLIndexKey) o;
int result = url.compareTo(other.getUrl());
if (result == 0) result = CUID.compare(userid, other.userid);
return result;
}
/**
* A Comparator optimized for IncrementalURLIndexKey.
*/
public static class GroupingComparator extends WritableComparator {
public GroupingComparator() {
super(IncrementalURLIndexKey.class, true);
}
public int compare(WritableComparable a, WritableComparable b) {
IncrementalURLIndexKey key1 = (IncrementalURLIndexKey) a;
IncrementalURLIndexKey key2 = (IncrementalURLIndexKey) b;
return key1.getUrl().compareTo(key2.getUrl());
}
}
static {
WritableComparator.define(IncrementalURLIndexKey.class, new
GroupingComparator());
}
}
Thanks,
Deepika
-----Original Message-----
From: Grant Ingersoll [mailto:gsingers@apache.org]
Sent: Monday, October 27, 2008 7:14 PM
To: core-user@hadoop.apache.org
Subject: Re: "Merge of the inmemory files threw an exception" and diffs
between 0.17.2 and 0.18.1
I suppose I should add an actual question: Is this a bug or are we
relying on a "feature" that has since been fixed?
On Oct 27, 2008, at 10:05 PM, Grant Ingersoll wrote:
> Hi,
>
> Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
> some of our clustering code and Hadoop 0.18.1. The thread in
> context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr
>
> The problem seems to occur when going from 0.17.2 to 0.18.1. In the
> user logs, we are seeing the following exception:
> 2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
> to the last merge-pass, with 2 segments left of total size: 5011 bytes
> 2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
> threw an exception: java.io.IOException: Intermedate merge failed
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $InMemFSMergeThread.run(ReduceTask.java:2078)
> Caused by: java.lang.NumberFormatException: For input string: "["
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
> 1224)
> at java.lang.Double.parseDouble(Double.java:510)
> at
> org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
> at
> org
> .apache
> .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
> at
> org
> .apache
> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
> 38)
> at
> org
> .apache
> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
> 31)
> at org.apache.hadoop.mapred.ReduceTask
> $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
> $3100(ReduceTask.java:341)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
>
> And in the main output log (from running bin/hadoop jar mahout/
> examples/build/apache-mahout-examples-0.1-dev.job
> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
> 08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
> attempt_200810272112_0011_r_000000_0, Status : FAILED
> java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
> copier failed
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
> at org.apache.hadoop.mapred.TaskTracker
> $Child.main(TaskTracker.java:2207)
>
> If I run this exact same job on 0.17.2 it all runs fine. I suppose
> either a bug was introduced in 0.18.1 or a bug was fixed that we
> were relying on. Looking at the release notes between the fixes,
> nothing in particular struck me as related. If it helps, I can
> provide the instructions for how to run the example in question
> (they need to be written up anyway!)
>
>
> I see some related things at
http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+a
n+exception
> , but those are older, it seems, so not sure what to make of them.
>
> Thanks,
> Grant
Re: "Merge of the inmemory files threw an exception" and diffs between 0.17.2 and 0.18.1
Posted by Grant Ingersoll <gs...@apache.org>.
I suppose I should add an actual question: Is this a bug or are we
relying on a "feature" that has since been fixed?
On Oct 27, 2008, at 10:05 PM, Grant Ingersoll wrote:
> Hi,
>
> Over in Mahout (lucene.a.o/mahout), we are seeing an oddity with
> some of our clustering code and Hadoop 0.18.1. The thread in
> context is at: http://mahout.markmail.org/message/vcyvlz2met7fnthr
>
> The problem seems to occur when going from 0.17.2 to 0.18.1. In the
> user logs, we are seeing the following exception:
> 2008-10-27 21:18:37,014 INFO org.apache.hadoop.mapred.Merger: Down
> to the last merge-pass, with 2 segments left of total size: 5011 bytes
> 2008-10-27 21:18:37,033 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200810272112_0011_r_000000_0 Merge of the inmemory files
> threw an exception: java.io.IOException: Intermedate merge failed
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2147)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $InMemFSMergeThread.run(ReduceTask.java:2078)
> Caused by: java.lang.NumberFormatException: For input string: "["
> at
> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:
> 1224)
> at java.lang.Double.parseDouble(Double.java:510)
> at
> org.apache.mahout.matrix.DenseVector.decodeFormat(DenseVector.java:60)
> at
> org
> .apache
> .mahout.matrix.AbstractVector.decodeVector(AbstractVector.java:256)
> at
> org
> .apache
> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
> 38)
> at
> org
> .apache
> .mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:
> 31)
> at org.apache.hadoop.mapred.ReduceTask
> $ReduceCopier.combineAndSpill(ReduceTask.java:2174)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access
> $3100(ReduceTask.java:341)
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> $InMemFSMergeThread.doInMemMerge(ReduceTask.java:2134)
>
> And in the main output log (from running bin/hadoop jar mahout/
> examples/build/apache-mahout-examples-0.1-dev.job
> org.apache.mahout.clustering.syntheticcontrol.kmeans.Job) we see:
> 08/10/27 21:18:41 INFO mapred.JobClient: Task Id :
> attempt_200810272112_0011_r_000000_0, Status : FAILED
> java.io.IOException: attempt_200810272112_0011_r_000000_0The reduce
> copier failed
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
> at org.apache.hadoop.mapred.TaskTracker
> $Child.main(TaskTracker.java:2207)
>
> If I run this exact same job on 0.17.2 it all runs fine. I suppose
> either a bug was introduced in 0.18.1 or a bug was fixed that we
> were relying on. Looking at the release notes between the fixes,
> nothing in particular struck me as related. If it helps, I can
> provide the instructions for how to run the example in question
> (they need to be written up anyway!)
>
>
> I see some related things at http://hadoop.markmail.org/search/?q=Merge+of+the+inmemory+files+threw+an+exception
> , but those are older, it seems, so not sure what to make of them.
>
> Thanks,
> Grant