Choreographing a Kryo update

[Steve Loughran <st...@hortonworks.com>]

Spark is currently on a fairly dated version of Kryo 2.x; it's trailing on the fixes in Hive and, as the APIs are incompatible, resulted in that mutant spark-project/hive JAR needed for the Hive 1.2.1 support

But: updating it hasn't been an option, because Spark needs to be in sync with Twitter's Chill library.

There's now an offer from Twitter to help coordinate a kryo update across Chill, Scalding and other things they use

https://github.com/twitter/chill/pull/230

Given kryo is "The guava jar of serialization", I doubt anyone is jumping up and down wanting this, but it is something to consider. Once hive moves to it, all the hive spark integration is probably going to break again; getting in sync with hive (see SPARK-10793) would reduce the traumaticness of hive updates

Re: Choreographing a Kryo update

[Reynold Xin <rx...@databricks.com>]

We should consider this for Spark 2.0.


On Wed, Nov 11, 2015 at 2:01 PM, Steve Loughran <st...@hortonworks.com>
wrote:

>
>
> Spark is currently on a fairly dated version of Kryo 2.x; it's trailing on
> the fixes in Hive and, as the APIs are incompatible, resulted in that
> mutant spark-project/hive JAR needed for the Hive 1.2.1 support
>
> But: updating it hasn't been an option, because Spark needs to be in sync
> with Twitter's Chill library.
>
> There's now an offer from Twitter to help coordinate a kryo update across
> Chill, Scalding and other things they use
>
> https://github.com/twitter/chill/pull/230
>
> Given kryo is "The guava jar of serialization", I doubt anyone is jumping
> up and down wanting this, but it is something to consider. Once hive moves
> to it, all the hive spark integration is probably going to break again;
> getting in sync with hive (see SPARK-10793) would reduce the traumaticness
> of hive updates
>