(spark) branch master updated: [SPARK-51198][CORE][DOCS] Revise `defaultMinPartitions` function description

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 920486dcc099 [SPARK-51198][CORE][DOCS] Revise `defaultMinPartitions` function description
920486dcc099 is described below

commit 920486dcc09909f492f5352aaa59ab5fd181fd19
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Feb 13 18:30:27 2025 +0800

    [SPARK-51198][CORE][DOCS] Revise `defaultMinPartitions` function description
    
    ### What changes were proposed in this pull request?
    
    This PR aims to revise the function description of `SparkContext.defaultMinPartitions`.
    
    ### Why are the changes needed?
    
    The repository, `https://github.com/mesos/spark`, was archived 11 years ago.
    
    We had better embed the description instead of pointing to the ancient repository. It's hard to figure out the real reasons from a series of comments.
    
    https://github.com/apache/spark/blob/0a102327d52712f3947cb90da671ffb18be62626/core/src/main/scala/org/apache/spark/SparkContext.scala#L2880
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a change of comment.
    
    ### How was this patch tested?
    
    Manual review.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #49929 from dongjoon-hyun/SPARK-51198.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Kent Yao <ya...@apache.org>
---
 core/src/main/scala/org/apache/spark/SparkContext.scala | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 143a0d5e633c..1804b862fd69 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -2877,7 +2877,10 @@ class SparkContext(config: SparkConf) extends Logging {
   /**
    * Default min number of partitions for Hadoop RDDs when not given by user
    * Notice that we use math.min so the "defaultMinPartitions" cannot be higher than 2.
-   * The reasons for this are discussed in https://github.com/mesos/spark/pull/718
+   * For large files, the Hadoop InputFormat library always creates more partitions even though
+   * defaultMinPartitions is 2. For small files, it can be good to process small files quickly.
+   * However, usually when Spark joins a small table with a big one, we'll still spend most of
+   * time on the map part of the big one anyway.
    */
   def defaultMinPartitions: Int = math.min(defaultParallelism, 2)
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org