Spark partition formula on standalone mode?

["Kali.tummala@gmail.com" <Ka...@gmail.com>]

Hi All,

I did worked on spark installed on Hadoop cluster but never worked on spark
on standalone cluster.

My question how to set number of partitions in spark when it's running on
spark standalone cluster?

If spark on Hadoop I calculate my formula using hdfs block sizes but how I
calculate without hdfs block size if spark running on standalone non Hadoop
cluster.
 
Partition formula for 100gb file:-
Hdfs block size:-256

100*1024  =400 partitions
/256

Executors:- 100/4= 25

Executor memory:- 160gb/25=7







--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-partition-formula-on-standalone-mode-tp27237.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org