You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2015/10/14 15:21:05 UTC
[jira] [Comment Edited] (CRUNCH-571) Scrunch functions fail
serialization check in the REPL
[ https://issues.apache.org/jira/browse/CRUNCH-571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14956912#comment-14956912 ]
Tom White edited comment on CRUNCH-571 at 10/14/15 1:20 PM:
------------------------------------------------------------
To reproduce run
{code}
import org.apache.crunch.scrunch.Mem
case class A(x: Int)
val pColOfA = Mem.collectionOf(A(1), A(2))
pColOfA.filter(row => row.x == 1)
{code}
which results in
{noformat}
java.lang.IllegalStateException: anon$2 named 'null' cannot be serialized
at org.apache.crunch.impl.mem.collect.MemCollection.verifySerializable(MemCollection.java:110)
at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:129)
at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:123)
at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:118)
at org.apache.crunch.impl.mem.collect.MemCollection.filter(MemCollection.java:275)
at org.apache.crunch.scrunch.PCollectionLike$class.filter(PCollectionLike.scala:106)
at org.apache.crunch.scrunch.PCollection.filter(PCollection.scala:29)
... 33 elided
Caused by: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: $anonfun$1
at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:165)
at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:192)
at org.apache.crunch.impl.mem.collect.MemCollection.verifySerializable(MemCollection.java:108)
... 39 more
Caused by: java.lang.ClassNotFoundException: $anonfun$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:625)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:162)
... 41 more
{noformat}
was (Author: tomwhite):
To reproduce run
{code}
import org.apache.crunch.scrunch.Mem
case class A(x: Int)
val pColOfA = Mem.collectionOf(A(1), A(2))
pColOfA.filter(row => row.x == 1)
{code}
which results in
{noformat}
java.lang.IllegalStateException: anon$2 named 'null' cannot be serialized
at org.apache.crunch.impl.mem.collect.MemCollection.verifySerializable(MemCollection.java:115)
at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:177)
at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:171)
at org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:166)
at org.apache.crunch.impl.mem.collect.MemCollection.filter(MemCollection.java:323)
at org.apache.crunch.scrunch.PCollectionLike$class.filter(PCollectionLike.scala:106)
at org.apache.crunch.scrunch.PCollection.filter(PCollection.scala:29)
... 33 elided
Caused by: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: $anonfun$1
at org.apache.crunch.impl.mem.collect.MemCollection.deserialize(MemCollection.java:142)
at org.apache.crunch.impl.mem.collect.MemCollection.deserialize(MemCollection.java:161)
at org.apache.crunch.impl.mem.collect.MemCollection.verifySerializable(MemCollection.java:113)
... 39 more
Caused by: java.lang.ClassNotFoundException: $anonfun$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:625)
at org.apache.crunch.impl.mem.collect.MemCollection$1.resolveClass(MemCollection.java:135)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.crunch.impl.mem.collect.MemCollection.deserialize(MemCollection.java:139)
... 41 more
{noformat}
> Scrunch functions fail serialization check in the REPL
> ------------------------------------------------------
>
> Key: CRUNCH-571
> URL: https://issues.apache.org/jira/browse/CRUNCH-571
> Project: Crunch
> Issue Type: Bug
> Components: Scrunch
> Affects Versions: 0.12.0
> Reporter: Tom White
> Assignee: Tom White
> Attachments: CRUNCH-571.patch
>
>
> Calling {{pcol.filter(row => row.x == 1)}} results in {{java.lang.IllegalStateException: anon$2 named 'null' cannot be serialized}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)