You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2018/01/17 15:15:04 UTC
[jira] [Commented] (SPARK-23131) Stackoverflow using ML and Kryo
serializer
[ https://issues.apache.org/jira/browse/SPARK-23131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328868#comment-16328868 ]
Sean Owen commented on SPARK-23131:
-----------------------------------
This requires updating Twitter Chill too, really, to 0.9.2. Have you tried it? I am running tests now and looks pretty good.
Although that would mean a major version update to Kryo, I don't think we guarantee any interoperability at runtime across minor releases of Spark, so any change in its formats (and they changed) could be acceptable for Spark 2.4.0. Certainly for 3.0.
Kryo 3.0.3 is almost 3 years old now so yeah may be time.
> Stackoverflow using ML and Kryo serializer
> ------------------------------------------
>
> Key: SPARK-23131
> URL: https://issues.apache.org/jira/browse/SPARK-23131
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 2.2.0
> Reporter: Peigen
> Priority: Minor
>
> When trying to use GeneralizedLinearRegression model and set SparkConf to use KryoSerializer(JavaSerializer is fine)
> It causes StackOverflowException
> {quote}
> Exception in thread "dispatcher-event-loop-34" java.lang.StackOverflowError
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.get(HashMap.java:556)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:61)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:62)
> {quote}
> This is very likely to be https://github.com/EsotericSoftware/kryo/issues/341
> Upgrade Kryo to 4.0+ probably could fix this
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org