[jira] [Commented] (SPARK-7483) [MLLib] Using Kryo with FPGrowth fails with an exception

["Joseph K. Bradley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535562#comment-14535562 ] 

Joseph K. Bradley commented on SPARK-7483:
------------------------------------------

I believe this is because it would need a custom serializer.  Not all classes in Spark work with Kryo out of the box.  But if you want to learn more and write your own, please check out: [http://spark.apache.org/docs/latest/tuning.html#data-serialization]

Also, this kind of question should probably go to the user list before JIRA.  I'll close this, but if you think I'm wrong, please bring up the issue on the user list!  Thanks

> [MLLib] Using Kryo with FPGrowth fails with an exception
> --------------------------------------------------------
>
>                 Key: SPARK-7483
>                 URL: https://issues.apache.org/jira/browse/SPARK-7483
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>    Affects Versions: 1.3.1
>            Reporter: Tomasz Bartczak
>            Priority: Minor
>
> When using FPGrowth algorithm with KryoSerializer - Spark fails with
> {code}
> Job aborted due to stage failure: Task 0 in stage 9.0 failed 1 times, most recent failure: Lost task 0.0 in stage 9.0 (TID 16, localhost): com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Can not set final scala.collection.mutable.ListBuffer field org.apache.spark.mllib.fpm.FPTree$Summary.nodes to scala.collection.mutable.ArrayBuffer
> Serialization trace:
> nodes (org.apache.spark.mllib.fpm.FPTree$Summary)
> org$apache$spark$mllib$fpm$FPTree$$summaries (org.apache.spark.mllib.fpm.FPTree)
> {code}
> This can be easily reproduced in spark codebase by setting 
> {code}
> conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
> {code} and running FPGrowthSuite.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org