[jira] [Created] (SPARK-27799) Allow SerializerManager.canUseKryo to be customized via configuration

["Josh Rosen (JIRA)" <ji...@apache.org>]

Josh Rosen created SPARK-27799:
----------------------------------

             Summary: Allow SerializerManager.canUseKryo to be customized via configuration
                 Key: SPARK-27799
                 URL: https://issues.apache.org/jira/browse/SPARK-27799
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.4.0
            Reporter: Josh Rosen


Kryo serialization can offer a substantial performance boost compared to Java serialization and I generally recommend that users configure Spark to use it.

In general it may not be safe to blindly flip the default to Kryo: certain jobs might depend on Java serialization, so switching them to Kryo might cause crashes or incorrect behavior.

However, we may know that certain data types are safe to serialize with Kryo, in which case we can whitelist _just those types_ for use with Kryo serialization but keep everything else using the default Java serializer.

Back in SPARK-13926 (Spark 2.0) I added a {{SerializerManager}} to implement this idea for strings, primitives, primitive arrays, and a few other data types: those types will automatically use Kryo serialization when at top-level types in RDDs. However, there's no ability for users to customize / extend this whitelist.

I propose to add a new user-facing configuration, name TBD, which accepts a comma-separated list of class / interface names and uses the to expand the {{SerializerMananger.canUseKryo}} whitelist.

This will allow advanced users to incrementally default to Kryo for certain types (e.g. Scrooge ThriftStructs).

This feature is useful for "data platform" teams who provide Spark-as-a-service to internal customers: with this proposed configuration, platform teams can configure global defaults for serialization in a way which is more incremental / narrow-in-scope than simply defaulting to Kryo everywhere.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org