objectFile uses only java serializer?

[Koert Kuipers <ko...@tresata.com>]

in the comments on SparkContext.objectFile it says:
"It will also be pretty slow if you use the default serializer (Java
serialization)"

this suggests the spark.serializer is used, which means i can switch to the
much faster kryo serializer. however when i look at the code it uses
Utils.deserialize, which is always using Java serialization.

did i get that right? and is this desired?
it seems straightforward to switch objectFile to use the serializer as
specified by spark.serializer (although it might being in new classloader
issues).

Re: objectFile uses only java serializer?

[Ted Yu <yu...@gmail.com>]

bq. it uses Utils.deserialize, which is always using Java serialization.

I agree with your finding.

On Mon, Mar 23, 2015 at 1:14 PM, Koert Kuipers <ko...@tresata.com> wrote:

> in the comments on SparkContext.objectFile it says:
> "It will also be pretty slow if you use the default serializer (Java
> serialization)"
>
> this suggests the spark.serializer is used, which means i can switch to
> the much faster kryo serializer. however when i look at the code it uses
> Utils.deserialize, which is always using Java serialization.
>
> did i get that right? and is this desired?
> it seems straightforward to switch objectFile to use the serializer as
> specified by spark.serializer (although it might being in new classloader
> issues).
>