You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com on 2015/07/09 03:50:03 UTC
Spark program throws NIO Buffer over flow error (TDigest - Ted
Dunning lib)
Any suggestions ?
Code:
val dimQuantiles = genericRecordsAndKeys
.map {
case (keyToOutput, rec) =>
var digest: TDigest = TDigest.createAvlTreeDigest(10000)
val fpPaidGMB = rec.get("fpPaidGMB").asInstanceOf[Double]
digest.add(fpPaidGMB)
var bbuf: ByteBuffer = ByteBuffer.allocate(digest.byteSize());
digest.asBytes(bbuf);
(keyToOutput.toString, bbuf.array())
}
Trace:
aborted due to stage failure: Task 4487 in stage 1.0 failed 4 times, most
recent failure: Lost task 4487.3 in stage 1.0 (TID 281,
phxaishdc9dn0111.phx.ebay.com): java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Buffer.java:519)
at java.nio.HeapByteBuffer.putDouble(HeapByteBuffer.java:519)
at com.tdunning.math.stats.AVLTreeDigest.asBytes(AVLTreeDigest.java:336)
at
com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:96)
at
com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:89)
at
org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:197)
at
org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:196)
at
org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:138)
at
org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:202)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:56)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
--
Deepak
Re: Spark program throws NIO Buffer over flow error (TDigest - Ted
Dunning lib)
Posted by Ted Yu <yu...@gmail.com>.
Doesn't seem to be Spark problem, assuming TDigest comes from mahout.
Cheers
On Wed, Jul 8, 2015 at 7:49 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> Same exception with different values of compression (10,100)
>
> var digest: TDigest = TDigest.createAvlTreeDigest(100)
>
> On Wed, Jul 8, 2015 at 6:50 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
>> Any suggestions ?
>> Code:
>>
>> val dimQuantiles = genericRecordsAndKeys
>>
>> .map {
>>
>> case (keyToOutput, rec) =>
>>
>> var digest: TDigest = TDigest.createAvlTreeDigest(10000)
>>
>> val fpPaidGMB = rec.get("fpPaidGMB").asInstanceOf[Double]
>>
>> digest.add(fpPaidGMB)
>>
>> var bbuf: ByteBuffer = ByteBuffer.allocate(digest.byteSize());
>>
>> digest.asBytes(bbuf);
>>
>> (keyToOutput.toString, bbuf.array())
>>
>> }
>>
>>
>> Trace:
>> aborted due to stage failure: Task 4487 in stage 1.0 failed 4 times, most
>> recent failure: Lost task 4487.3 in stage 1.0 (TID 281,
>> phxaishdc9dn0111.phx.ebay.com): java.nio.BufferOverflowException
>> at java.nio.Buffer.nextPutIndex(Buffer.java:519)
>> at java.nio.HeapByteBuffer.putDouble(HeapByteBuffer.java:519)
>> at com.tdunning.math.stats.AVLTreeDigest.asBytes(AVLTreeDigest.java:336)
>> at
>> com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:96)
>> at
>> com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:89)
>> at
>> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:197)
>> at
>> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:196)
>> at
>> org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:138)
>> at
>> org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32)
>> at
>> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:202)
>> at
>> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:56)
>> at
>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
>> at
>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>>
>>
>> --
>> Deepak
>>
>>
>
>
> --
> Deepak
>
>
Re: Spark program throws NIO Buffer over flow error (TDigest - Ted
Dunning lib)
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
Same exception with different values of compression (10,100)
var digest: TDigest = TDigest.createAvlTreeDigest(100)
On Wed, Jul 8, 2015 at 6:50 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> Any suggestions ?
> Code:
>
> val dimQuantiles = genericRecordsAndKeys
>
> .map {
>
> case (keyToOutput, rec) =>
>
> var digest: TDigest = TDigest.createAvlTreeDigest(10000)
>
> val fpPaidGMB = rec.get("fpPaidGMB").asInstanceOf[Double]
>
> digest.add(fpPaidGMB)
>
> var bbuf: ByteBuffer = ByteBuffer.allocate(digest.byteSize());
>
> digest.asBytes(bbuf);
>
> (keyToOutput.toString, bbuf.array())
>
> }
>
>
> Trace:
> aborted due to stage failure: Task 4487 in stage 1.0 failed 4 times, most
> recent failure: Lost task 4487.3 in stage 1.0 (TID 281,
> phxaishdc9dn0111.phx.ebay.com): java.nio.BufferOverflowException
> at java.nio.Buffer.nextPutIndex(Buffer.java:519)
> at java.nio.HeapByteBuffer.putDouble(HeapByteBuffer.java:519)
> at com.tdunning.math.stats.AVLTreeDigest.asBytes(AVLTreeDigest.java:336)
> at
> com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:96)
> at
> com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:89)
> at
> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:197)
> at
> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:196)
> at
> org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:138)
> at
> org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32)
> at
> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:202)
> at
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:56)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>
>
> --
> Deepak
>
>
--
Deepak