You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Peter Halliday <pj...@cornell.edu> on 2016/06/08 19:45:29 UTC
UnsupportedOperationException: converting from RDD to DataSets on 1.6.1
I have some code that was producing OOM during shuffle and was RDD. So, upon direction by a member of Databricks I started covering to Datasets. However, when we did we are getting an error that seems to be not liking something within one of our case classes.
Peter Halliday
[2016-06-08 19:12:22,083] ERROR org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - User class threw exception: java.lang.UnsupportedOperationException: No Encoder found for Set[com.wix.accord.Violation]
- field (class: "scala.collection.immutable.Set", name: "violations")
- root class: "com.here.probe.ingestion.converters.ProbeValidation"
java.lang.UnsupportedOperationException: No Encoder found for Set[com.wix.accord.Violation]
- field (class: "scala.collection.immutable.Set", name: "violations")
- root class: "com.here.probe.ingestion.converters.ProbeValidation"
at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:594)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:494)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:490)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:490)
at org.apache.spark.sql.catalyst.ScalaReflection$.extractorsFor(ScalaReflection.scala:402)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54)
at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:41)
at com.here.probe.ingestion.IngestProbe.processLines(IngestProbe.scala:116)
at com.here.probe.ingestion.IngestProbe.processFiles(IngestProbe.scala:86)
at com.here.probe.ingestion.IngestProbe$.main(IngestProbe.scala:53)
at com.here.probe.ingestion.IngestProbe.main(IngestProbe.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
[2016-06-08 19:12:22,086] INFO org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.UnsupportedOperationException: No Encoder found for Set[com.wix.accord.Violation]
- field (class: "scala.collection.immutable.Set", name: "violations")
- root class: "com.here.probe.ingestion.converters.ProbeValidation”)
Re: UnsupportedOperationException: converting from RDD to DataSets on 1.6.1
Posted by Koert Kuipers <ko...@tresata.com>.
You can try passing in an explicit encoder:
org.apache.spark.sql.Encoders.kryo[Set[com.wix.accord.Violation]]
Although this might only be available in spark 2, i don't remember top of
my head...
On Wed, Jun 8, 2016 at 11:57 PM, Koert Kuipers <ko...@tresata.com> wrote:
> Sets are not supported. you basically need to stick to products (tuples,
> case classes), Seq and Map (and in spark 2 also Option).
>
> Or you can need to resort to the kryo-based encoder.
>
> On Wed, Jun 8, 2016 at 3:45 PM, Peter Halliday <pj...@cornell.edu> wrote:
>
>> I have some code that was producing OOM during shuffle and was RDD. So,
>> upon direction by a member of Databricks I started covering to Datasets.
>> However, when we did we are getting an error that seems to be not liking
>> something within one of our case classes.
>>
>> Peter Halliday
>>
>>
>> [2016-06-08 19:12:22,083] ERROR
>> org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - User class
>> threw exception: java.lang.UnsupportedOperationException: No Encoder found
>> for Set[com.wix.accord.Violation]
>> - field (class: "scala.collection.immutable.Set", name: "violations")
>> - root class: "com.here.probe.ingestion.converters.ProbeValidation"
>> java.lang.UnsupportedOperationException: No Encoder found for
>> Set[com.wix.accord.Violation]
>> - field (class: "scala.collection.immutable.Set", name: "violations")
>> - root class: "com.here.probe.ingestion.converters.ProbeValidation"
>> at
>> org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:594)
>> at
>> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:494)
>> at
>> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:490)
>> at
>> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
>> at
>> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
>> at scala.collection.immutable.List.foreach(List.scala:318)
>> at
>> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
>> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
>> at
>> org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:490)
>> at
>> org.apache.spark.sql.catalyst.ScalaReflection$.extractorsFor(ScalaReflection.scala:402)
>> at
>> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54)
>> at
>> org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:41)
>> at
>> com.here.probe.ingestion.IngestProbe.processLines(IngestProbe.scala:116)
>> at com.here.probe.ingestion.IngestProbe.processFiles(IngestProbe.scala:86)
>> at com.here.probe.ingestion.IngestProbe$.main(IngestProbe.scala:53)
>> at com.here.probe.ingestion.IngestProbe.main(IngestProbe.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
>> [2016-06-08 19:12:22,086] INFO
>> org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - Final app
>> status: FAILED, exitCode: 15, (reason: User class threw exception:
>> java.lang.UnsupportedOperationException: No Encoder found for
>> Set[com.wix.accord.Violation]
>> - field (class: "scala.collection.immutable.Set", name: "violations")
>> - root class: "com.here.probe.ingestion.converters.ProbeValidation”)
>>
>
>
Re: UnsupportedOperationException: converting from RDD to DataSets on 1.6.1
Posted by Koert Kuipers <ko...@tresata.com>.
Sets are not supported. you basically need to stick to products (tuples,
case classes), Seq and Map (and in spark 2 also Option).
Or you can need to resort to the kryo-based encoder.
On Wed, Jun 8, 2016 at 3:45 PM, Peter Halliday <pj...@cornell.edu> wrote:
> I have some code that was producing OOM during shuffle and was RDD. So,
> upon direction by a member of Databricks I started covering to Datasets.
> However, when we did we are getting an error that seems to be not liking
> something within one of our case classes.
>
> Peter Halliday
>
>
> [2016-06-08 19:12:22,083] ERROR
> org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - User class
> threw exception: java.lang.UnsupportedOperationException: No Encoder found
> for Set[com.wix.accord.Violation]
> - field (class: "scala.collection.immutable.Set", name: "violations")
> - root class: "com.here.probe.ingestion.converters.ProbeValidation"
> java.lang.UnsupportedOperationException: No Encoder found for
> Set[com.wix.accord.Violation]
> - field (class: "scala.collection.immutable.Set", name: "violations")
> - root class: "com.here.probe.ingestion.converters.ProbeValidation"
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:594)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:494)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:490)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:490)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.extractorsFor(ScalaReflection.scala:402)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54)
> at
> org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:41)
> at com.here.probe.ingestion.IngestProbe.processLines(IngestProbe.scala:116)
> at com.here.probe.ingestion.IngestProbe.processFiles(IngestProbe.scala:86)
> at com.here.probe.ingestion.IngestProbe$.main(IngestProbe.scala:53)
> at com.here.probe.ingestion.IngestProbe.main(IngestProbe.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
> [2016-06-08 19:12:22,086] INFO
> org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - Final app
> status: FAILED, exitCode: 15, (reason: User class threw exception:
> java.lang.UnsupportedOperationException: No Encoder found for
> Set[com.wix.accord.Violation]
> - field (class: "scala.collection.immutable.Set", name: "violations")
> - root class: "com.here.probe.ingestion.converters.ProbeValidation”)
>