You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by David Thomas <dt...@gmail.com> on 2014/02/16 22:22:24 UTC

Setting serializer in Spark shell

How can I set the default serializer to Kyro when using Spark Shell?

Re: Setting serializer in Spark shell

Posted by suman bharadwaj <su...@gmail.com>.

https://groups.google.com/forum/#!msg/spark-users/bXXG3-JHXHk/sU6oFPfg_kcJ

See Matei's answer.

Regards,
SB


On Mon, Feb 17, 2014 at 2:52 AM, David Thomas <dt...@gmail.com> wrote:

> How can I set the default serializer to Kyro when using Spark Shell?
>

Re: Setting serializer in Spark shell

Posted by Nan Zhu <zh...@gmail.com>.

try

OPTIONS="-Dspark.serializer=xxxx" bin/spark-shell

but you cannot use -c option, otherwise it will overwrite your external OPTIONS setup

if you have to set -c , you have to modify your bin/spark-shell a bit, before

for o in "$@"; do

add  

OPTIONS="-Dspark.serializer=xxxx”

P.S. I didn’t try by myself, but I just modified something similar, I guess it will work

Best,  

--  
Nan Zhu



On Sunday, February 16, 2014 at 4:22 PM, David Thomas wrote:

> How can I set the default serializer to Kyro when using Spark Shell?