Value of SHUFFLE_PARTITIONS

[Chirag Aggarwal <Ch...@guavus.com>]

Hi,

Currently the number of shuffle partitions is config driven parameter (SHUFFLE_PARTITIONS) . This means that anyone who is running a spark-sql query should first of
all analyze that what value of SHUFFLE_PARTITIONS would give the best performance for the query.

Shouldn't there be a logic in SparkSql which should be able to figure out the best value and also provide a mechanism to give preference to user specified value.
This I believe can be worked out on the basis of number of partitions in the original data.

I ran some queries and with default value (200) of shuffle-partitioning, and when I changed this value to 5, the time taken by the query reduced by nearly 35%.

Thanks,
Chirag