[jira] [Created] (SPARK-15831) Kryo 2.21 TreeMap serialization bug causes random job failures with RDDs of HBase puts

["Charles Gariépy-Ikeson (JIRA)" <ji...@apache.org>]

Charles Gariépy-Ikeson created SPARK-15831:
----------------------------------------------

             Summary: Kryo 2.21 TreeMap serialization bug causes random job failures with RDDs of HBase puts
                 Key: SPARK-15831
                 URL: https://issues.apache.org/jira/browse/SPARK-15831
             Project: Spark
          Issue Type: Bug
            Reporter: Charles Gariépy-Ikeson


This was found on Spark 1.5, but it seems that all Spark 1.x brings in the problematic dependency in question.

Kryo 2.21 has a bug when serializing TreeMap that causes intermittent failures in Spark. This problem cause be seen especially when sinking data to HBase using a RDD of HBase Puts (which internally have TreeMap).

Kryo fixed the issue in 2.21.1. Current work around involves setting "spark.kryo.referenceTracking" to false.

For reference see:
Kryo commit: https://github.com/EsotericSoftware/kryo/commit/00ffc7ed443e022a8438d1e4c4f5b86fe4f9912b
TreeMap Kryo Issue: https://github.com/EsotericSoftware/kryo/issues/112
HBase Put Kryo Issue: https://github.com/EsotericSoftware/kryo/issues/428



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org