You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by cjrumble <cj...@gmail.com> on 2015/12/08 23:22:27 UTC
spark-defaults.conf optimal configuration
I am seeking help with a Spark configuration running queries against a
cluster of 6 machines. Each machine has Spark 1.5.1 with slaves started on 6
and 1 acting as master/thriftserver. I query from Beeline 2 tables that have
300M and 31M rows respectively. Results from my queries thus far return up
to 500M rows when queried using Oracle but Spark errors at anything more
than 5.5M rows.
I believe there is an optimal memory configuration that must be set for each
of the workers in our cluster but I have not been able to determine that
setting. Is there something better than trial and error? Are there settings
to avoid such as making sure not to set spark.driver.maxResultSize >
spark.driver.memory?
Is there a formula or guidelines by which to calculate the correct Spark
configuration values when given a machines available cores and memory
resources?
This is my current configuration:
BDA v3 server : SUN SERVER X4-2L
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
CPU cores : 32
GB of memory (>=63): 63
number of disks : 12 spark-defaults.conf
spark.driver.memory 20g
spark.executor.memory 40g
spark.executor.extraJavaOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
spark.rpc.askTimeout 6000s
spark.rpc.lookupTimeout 3000s
spark.driver.maxResultSize 20g
spark.rdd.compress true
spark.storage.memoryFraction 1
spark.core.connection.ack.wait.timeout 600
spark.akka.frameSize 500
spark.shuffle.compress true
spark.shuffle.file.buffer 128k
spark.shuffle.memoryFraction 0
spark.shuffle.spill.compress true
spark.shuffle.spill true
Thank you,
Chris
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-defaults-conf-optimal-configuration-tp25641.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark-defaults.conf optimal configuration
Posted by cjrumble <cj...@gmail.com>.
Hello Neelesh,
Thank you for the checklist for determining the correct configuration of
Spark. I will go through these and let you know if I have further questions.
Regards,
Chris
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-defaults-conf-optimal-configuration-tp25641p25649.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark-defaults.conf optimal configuration
Posted by nsalian <ns...@cloudera.com>.
Hi Chris,
Thank you for posting the question.
Tuning spark configurations is a tricky task since there are a lot factors
to consider.
The configurations that you listed cover the most them.
To understand the situation that can guide you in making a decision about
tuning:
1) What kind of spark applications are you intending to run?
2) What cluster manager have you decided to go with?
3) How frequent are these applications going to run? (For the sake of
scheduling)
4) Is this used by multiple users?
5) What else do you have in the cluster that will interact with Spark? (For
the sake of resolving dependencies)
Personally, I would suggest to have these questions prior to jumping on the
idea of tuning.
A cluster manager like YARN would help understand the settings for cores and
memory since the applications have to be considered for scheduling.
Hope that helps to start off in the right direction.
-----
Neelesh S. Salian
Cloudera
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-defaults-conf-optimal-configuration-tp25641p25642.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org