(spark) branch branch-4.x updated: [SPARK-60383][CORE] Keep streaming shuffle implementation details out of the public API

[ru...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 88c6a72df60a [SPARK-60383][CORE] Keep streaming shuffle implementation details out of the public API
88c6a72df60a is described below

commit 88c6a72df60aa1dcdb6122bc8c7c6a236d2042f9
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Thu Oct 22 18:50:21 2026 +0800

    [SPARK-60383][CORE] Keep streaming shuffle implementation details out of the public API
    
    ### What changes were proposed in this pull request?
    
    Mark three streaming shuffle implementation APIs as `private[spark]`:
    
    - `ShuffleLocationResponse`
    - `BarrierTaskContext.addPostStatusUpdateListener`
    - `SparkEnv.streamingShuffleOutputTracker`
    
    Please backport this fix to `branch-4.x` and `branch-4.3`.
    
    ### Why are the changes needed?
    
    These public declarations expose internal types that are already `private[spark]`. Restricting their visibility before the 4.3.0 release avoids establishing unintended public API commitments.
    
    ### Does this PR introduce _any_ user-facing change?
    
    The three internal declarations are no longer public Scala APIs. Their implementations and runtime behavior are unchanged.
    
    ### How was this patch tested?
    
    All 62 tests passed across `TaskContextSuite`, `BarrierTaskContextSuite`, and `StreamingShuffleOutputTrackerSuite` using `core/testOnly` with Java 17. The subsequent documentation and deprecation restoration was checked against the original source, and all three visibility fixes were verified to remain intact. `git diff --check` passed.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6)
    
    Closes #59633 from zhengruifeng/core-streaming-shuffle-api-visibility-dev-5.
    
    Authored-by: Ruifeng Zheng <ru...@apache.org>
    Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
    (cherry picked from commit 42230274f919789a86bbe1c37d5cb022c5294dec)
    Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
 core/src/main/scala/org/apache/spark/BarrierTaskContext.scala          | 3 ++-
 core/src/main/scala/org/apache/spark/SparkEnv.scala                    | 2 +-
 .../main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala    | 2 +-
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala b/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
index f9cf2c4bc0c9..cdbc01d3b183 100644
--- a/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
+++ b/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
@@ -302,7 +302,8 @@ class BarrierTaskContext private[spark] (
     taskContext.createResourceUninterruptibly(resourceBuilder)
   }
 
-  override def addPostStatusUpdateListener(listener: PostStatusUpdateListener): TaskContext = {
+  override private[spark] def addPostStatusUpdateListener(
+      listener: PostStatusUpdateListener): TaskContext = {
     taskContext.addPostStatusUpdateListener(listener)
     this
   }
diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala b/core/src/main/scala/org/apache/spark/SparkEnv.scala
index d6455ac3b4c5..81dea0c5a932 100644
--- a/core/src/main/scala/org/apache/spark/SparkEnv.scala
+++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala
@@ -456,7 +456,7 @@ class SparkEnv (
   @volatile private var _streamingShuffleOutputTracker: Option[StreamingShuffleOutputTracker] =
     None
 
-  def streamingShuffleOutputTracker: Option[StreamingShuffleOutputTracker] =
+  private[spark] def streamingShuffleOutputTracker: Option[StreamingShuffleOutputTracker] =
     _streamingShuffleOutputTracker
 
   /**
diff --git a/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala b/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
index df4e76b017db..7b62346d1f72 100644
--- a/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
+++ b/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
@@ -125,7 +125,7 @@ private[spark] class StreamingShuffleOutputTrackerMasterEndpoint(
   }
 }
 
-case class ShuffleLocationResponse(
+private[spark] case class ShuffleLocationResponse(
   shuffleTaskLocations: Map[Long, StreamingShuffleTaskLocation],
   numShuffleWriterTasks: Int)
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org