You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by tridib <tr...@live.com> on 2015/09/23 09:42:55 UTC
How to control spark.sql.shuffle.partitions per query
I am having GC issue with default value of spark.sql.shuffle.partitions
(200). When I increase it to 2000, shuffle join works fine.
I want to use different values for spark.sql.shuffle.partitions depending on
data volume, for different queries which are fired from sane SparkSql
context.
Thanks
Tridib
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-control-spark-sql-shuffle-partitions-per-query-tp24781.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to control spark.sql.shuffle.partitions per query
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at the following for example:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
Search for spark.sql.shuffle.partitions and SQLConf.SHUFFLE_PARTITIONS.key
FYI
On Wed, Sep 23, 2015 at 12:42 AM, tridib <tr...@live.com> wrote:
> I am having GC issue with default value of spark.sql.shuffle.partitions
> (200). When I increase it to 2000, shuffle join works fine.
>
> I want to use different values for spark.sql.shuffle.partitions depending
> on
> data volume, for different queries which are fired from sane SparkSql
> context.
>
> Thanks
> Tridib
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-control-spark-sql-shuffle-partitions-per-query-tp24781.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>