You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "vdiwakar.malladi" <vd...@gmail.com> on 2014/12/05 19:51:29 UTC
Optimized spark configuration
Hi
Could any one help what would be better / optimized configuration for driver
memory, worker memory, number of parallelisms etc., parameters to be
configured when we are running 1 master node (it itself acting as slave node
also) and 1 slave node. Both are of 32 GB RAM with 4 cores.
On this, I loaded approx. 17M rows of data (3.2 GB) to hive store and when I
try to execute a query on this from jdbc thrift server, it is taking about
10-12 sec to retrieve the data which I think is too much.
Or guide please guide me any tutorial which will explain about these
optimize configurations.
Thanks in advance.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Optimized-spark-configuration-tp20495.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
R: Optimized spark configuration
Posted by Paolo Platter <pa...@agilelab.it>.
What kind of Query are you performing?
You should set something like 2 partition per core that would be 400 Mb per partition.
As you have a lot of ram I suggest to cache the whole table, performance will increase a lot.
Paolo
Inviata dal mio Windows Phone
________________________________
Da: vdiwakar.malladi<ma...@gmail.com>
Inviato: 05/12/2014 18:52
A: user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>
Oggetto: Optimized spark configuration
Hi
Could any one help what would be better / optimized configuration for driver
memory, worker memory, number of parallelisms etc., parameters to be
configured when we are running 1 master node (it itself acting as slave node
also) and 1 slave node. Both are of 32 GB RAM with 4 cores.
On this, I loaded approx. 17M rows of data (3.2 GB) to hive store and when I
try to execute a query on this from jdbc thrift server, it is taking about
10-12 sec to retrieve the data which I think is too much.
Or guide please guide me any tutorial which will explain about these
optimize configurations.
Thanks in advance.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Optimized-spark-configuration-tp20495.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org