You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sparky <Gu...@bah.com> on 2014/07/22 15:01:13 UTC
collect() on small group of Avro files causes plain
NullPointerException
Running a simple collect method on a group of Avro objects causes a plain
NullPointerException. Does anyone know what may be wrong?
>files.collect()
Press ENTER or type command to continue
Exception in thread "Executor task launch worker-0"
java.lang.NullPointerException
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$2.apply(Executor.scala:254)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$2.apply(Executor.scala:254)
at scala.Option.flatMap(Option.scala:170)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:254)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/collect-on-small-group-of-Avro-files-causes-plain-NullPointerException-tp10400.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: collect() on small group of Avro files causes plain NullPointerException
Posted by Eugen Cepoi <ce...@gmail.com>.
Do you have a list/array in your avro record? If yes this could cause the
problem. I experienced this kind of problem and solved it by providing
custom kryo ser/de for avro lists. Also be carefull spark reuses records,
so if you just read and then don't copy/transform them you would end up
with the records having same values.
2014-07-22 15:01 GMT+02:00 Sparky <Gu...@bah.com>:
> Running a simple collect method on a group of Avro objects causes a plain
> NullPointerException. Does anyone know what may be wrong?
>
> >files.collect()
>
> Press ENTER or type command to continue
> Exception in thread "Executor task launch worker-0"
> java.lang.NullPointerException
> at
>
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$2.apply(Executor.scala:254)
> at
>
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$2.apply(Executor.scala:254)
> at scala.Option.flatMap(Option.scala:170)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:254)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/collect-on-small-group-of-Avro-files-causes-plain-NullPointerException-tp10400.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: collect() on small list causes NullPointerException
Posted by Sparky <Gu...@bah.com>.
For those curious I was using KryoRegistrator it was causing some null
pointer exception. I removed the code and problem went away.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/collect-on-small-list-causes-NullPointerException-tp10400p10402.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.