You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@mahout.apache.org by Pat Ferrel <pa...@occamsmachete.com> on 2015/02/20 01:26:14 UTC

intermitent unit test error

It seems like after a clean install I fail some test in spark like the one below. Then when I build again starting with -rf spark everything passes. I’m not sure its always after a clean but happens often.

Anyone running into this?

- C = A cbind B, cogroup *** FAILED ***
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 205.0 failed 1 times, most recent failure: Lost task 0.0 in stage 205.0 (TID 442, localhost): java.io.IOException: PARSING_ERROR(2)
        org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
        org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
        org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
        org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
        org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
        org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
        org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
        org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
        org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
        sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:606)
        java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
        org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
  at scala.Option.foreach(Option.scala:236)
  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
  ...
- C = A cbind B, zip

Re: intermitent unit test error

Posted by Pat Ferrel <pa...@occamsmachete.com>.

I wonder of it has to do with the number of contexts problem since it always completes with a partial build.

I’d like to get that in with 1.2.1 this weekend so maybe it will go away.

On Feb 20, 2015, at 10:06 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:

We had various operational configuration problems with snappy as well so
had to disable it for now completely until somebody has time to figure it
out (which has been like forever)

On Thu, Feb 19, 2015 at 4:26 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:

> It seems like after a clean install I fail some test in spark like the one
> below. Then when I build again starting with -rf spark everything passes.
> I’m not sure its always after a clean but happens often.
> 
> Anyone running into this?
> 
> - C = A cbind B, cogroup *** FAILED ***
>  org.apache.spark.SparkException: Job aborted due to stage failure: Task
> 0 in stage 205.0 failed 1 times, most recent failure: Lost task 0.0 in
> stage 205.0 (TID 442, localhost): java.io.IOException: PARSING_ERROR(2)
>        org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
>        org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
>        org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
> 
> org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
> 
> org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
> 
> org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
> 
> org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
> 
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
> 
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
>        sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        java.lang.reflect.Method.invoke(Method.java:606)
> 
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>        java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> 
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> 
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>        java.lang.Thread.run(Thread.java:745)
> Driver stacktrace:
>  at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
>  at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
>  at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
>  at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>  at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
>  at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
>  at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
>  at scala.Option.foreach(Option.scala:236)
>  at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
>  ...
> - C = A cbind B, zip
> 
>

Re: intermitent unit test error

Posted by Dmitriy Lyubimov <dl...@gmail.com>.

We had various operational configuration problems with snappy as well so
had to disable it for now completely until somebody has time to figure it
out (which has been like forever)

On Thu, Feb 19, 2015 at 4:26 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:

> It seems like after a clean install I fail some test in spark like the one
> below. Then when I build again starting with -rf spark everything passes.
> I’m not sure its always after a clean but happens often.
>
> Anyone running into this?
>
> - C = A cbind B, cogroup *** FAILED ***
>   org.apache.spark.SparkException: Job aborted due to stage failure: Task
> 0 in stage 205.0 failed 1 times, most recent failure: Lost task 0.0 in
> stage 205.0 (TID 442, localhost): java.io.IOException: PARSING_ERROR(2)
>         org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
>         org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
>         org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
>
> org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
>
> org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
>
> org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
>
> org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
>
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
>
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
>         sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         java.lang.reflect.Method.invoke(Method.java:606)
>
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
>
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
>
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
>
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:745)
> Driver stacktrace:
>   at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
>   at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
>   at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
>   at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
>   at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
>   at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
>   at scala.Option.foreach(Option.scala:236)
>   at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
>   ...
> - C = A cbind B, zip
>
>