You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Shuai Zheng <sz...@gmail.com> on 2015/03/10 20:19:39 UTC

How to pass parameter to spark-shell when choose client mode --master yarn-client

Hi All,

 

I try to pass parameter to the spark-shell when I do some test:

 

spark-shell --driver-memory 512M --executor-memory 4G --master
spark://****:7077 --conf spark.sql.parquet.compression.codec=snappy --conf
spark.sql.parquet.binaryAsString=true

 

This works fine on my local pc. And when I start EMR, and pass the similar
things:

 

~/spark/bin/spark-shell --executor-memory 40G --master yarn-client
--num-executors 1 --executor-cores 32 --conf
spark.sql.parquet.compression.codec=snappy --conf
spark.sql.parquet.binaryAsString=true --conf
spark.serializer=org.apache.spark.serializer.KryoSerializer

 

The parameter doesn't pass to spark-shell. Anyone knows why? And what is the
alternatives for me to do that?

 

Regards,

 

Shuai