Failed to persist models with java.lang.NegativeArraySizeException exception in kryo

[LiJunjie <ho...@gmail.com>]

Hi all,

I have successfully trained, but failed to persist models. I notice PredictionIO will serialize models with kryo into Array[Byte], then what will happen if the trained models are very big. I also noticed the driver JVM heap grow rapidly at that time.

I use the official e-commerce recommender template, and the model to be serialized like this:

 class ECommModel(
   val rank: Int,
   val userFeatures: Map[Int, Array[Double]],
   val productModels: Map[Int, ProductModel],
   val userStringIntMap: BiMap[String, Int],
   val itemStringIntMap: BiMap[String, Int]
 ) extends Serializable

But there are several millions of records in userFeatures and productModels!

Here is the backtrace:

[INFO] [Engine$] EngineWorkflow.train completed
[INFO] [Engine] engineInstanceId=9f31aafc-2d66-4b76-9995-ad3e9fb84b6b
[INFO] [CoreWorkflow$] Inserting persistent model


[INFO] [AbstractConnector] Stopped Spark@5d1f640c{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
Exception in thread "main" com.esotericsoftware.kryo.KryoException: java.lang.NegativeArraySizeException
Serialization trace:
userFeatures (org.example.ecommercerecommendation.ECommModel)
        at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518)
        at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
        at com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29)
        at com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27)
        at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21)
        at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
        at com.twitter.chill.SerDeState.writeClassAndObject(SerDeState.java:64)
        at com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:116)
        at com.twitter.chill.KryoInjectionInstance.apply(KryoInjection.scala:64)
        at com.twitter.chill.KryoInjectionInstance.apply(KryoInjection.scala:55)
        at org.apache.predictionio.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:81)
        at org.apache.predictionio.workflow.CreateWorkflow$.main(CreateWorkflow.scala:251)
        at org.apache.predictionio.workflow.CreateWorkflow.main(CreateWorkflow.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NegativeArraySizeException
        at com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectIntMap.java:447)
        at com.esotericsoftware.kryo.util.IdentityObjectIntMap.putStash(IdentityObjectIntMap.java:245)
        at com.esotericsoftware.kryo.util.IdentityObjectIntMap.push(IdentityObjectIntMap.java:239)
        at com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:135)
        at com.esotericsoftware.kryo.util.IdentityObjectIntMap.putStash(IdentityObjectIntMap.java:246)
        at com.esotericsoftware.kryo.util.IdentityObjectIntMap.push(IdentityObjectIntMap.java:239)
        at com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntMap.java:135)
        at com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapReferenceResolver.java:41)
        at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:658)
        at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:623)
        at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37)
        at com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
        at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
        at com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:29)
        at com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable.scala:27)
        at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at com.twitter.chill.TraversableSerializer.write(Traversable.scala:27)
        at com.twitter.chill.TraversableSerializer.write(Traversable.scala:21)
        at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
        at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80)
        ... 27 more