[jira] [Comment Edited] (SPARK-4630) Dynamically determine optimal number of partitions

["Mike Dusenberry (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563839#comment-15563839 ] 

Mike Dusenberry edited comment on SPARK-4630 at 10/10/16 11:10 PM:
-------------------------------------------------------------------

It would be really nice to revisit this issue, perhaps even updated to just focus on {{DataSet}} given the current direction of the project.  

Basically, I would be really interested in smart DataSet/DataFrame/RDD repartitioning that automatically decides the proper number of partitions based on characteristics of the data (i.e. width) and the cluster.  From my experience, the outcome of a "wrong" number of partitions is frequent OOM errors, 2GB partition limits (although that's been lifted in 2.x, it's still a perf issue), and if you save a DataFrame/DataSet to, say, Parquet format with too few partitions, the individual compressed files may be too large to read later on (say if a partition is 2GB, but there isn't enough executor memory to open that when working with the files later on with a different Spark setup -- think perhaps a batch preprocessing cluster vs. production serving cluster).


was (Author: mwdusenb@us.ibm.com):
It would be really nice to revisit this issue, perhaps even updated to just focus on {{DataSet}} given the current direction of the project.  

Basically, I would be really interested in Smart DataSet/DataFrame/RDD repartitioning that automatically decides the proper number of partitions based on characteristics of the data (i.e. width) and the cluster.  From my experience, the outcome of a "wrong" number of partitions is frequent OOM errors, 2GB partition limits (although that's been lifted in 2.x, it's still a perf issue), and if you save a DataFrame/DataSet to, say, Parquet format with too few partitions, the individual compressed files may be too large to read later on (say if a partition is 2GB, but there isn't enough executor memory to open that when working with the files later on with a different Spark setup -- think perhaps a batch preprocessing cluster vs. production serving cluster).

> Dynamically determine optimal number of partitions
> --------------------------------------------------
>
>                 Key: SPARK-4630
>                 URL: https://issues.apache.org/jira/browse/SPARK-4630
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Kostas Sakellis
>            Assignee: Kostas Sakellis
>
> Partition sizes play a big part in how fast stages execute during a Spark job. There is a direct relationship between the size of partitions to the number of tasks - larger partitions, fewer tasks. For better performance, Spark has a sweet spot for how large partitions should be that get executed by a task. If partitions are too small, then the user pays a disproportionate cost in scheduling overhead. If the partitions are too large, then task execution slows down due to gc pressure and spilling to disk.
> To increase performance of jobs, users often hand optimize the number(size) of partitions that the next stage gets. Factors that come into play are:
> Incoming partition sizes from previous stage
> number of available executors
> available memory per executor (taking into account spark.shuffle.memoryFraction)
> Spark has access to this data and so should be able to automatically do the partition sizing for the user. This feature can be turned off/on with a configuration option. 
> To make this happen, we propose modifying the DAGScheduler to take into account partition sizes upon stage completion. Before scheduling the next stage, the scheduler can examine the sizes of the partitions and determine the appropriate number tasks to create. Since this change requires non-trivial modifications to the DAGScheduler, a detailed design doc will be attached before proceeding with the work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org