You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2026/02/03 10:54:34 UTC
(spark) branch memory-stream-compat created (now 3ca4eb6f5577)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch memory-stream-compat
in repository https://gitbox.apache.org/repos/asf/spark.git
at 3ca4eb6f5577 [SPARK-53656][SS][FOLLOWUP] Remove confusing MemoryStream factory method
This branch includes the following new commits:
new 3ca4eb6f5577 [SPARK-53656][SS][FOLLOWUP] Remove confusing MemoryStream factory method
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 01/01: [SPARK-53656][SS][FOLLOWUP] Remove confusing MemoryStream factory method
Posted by we...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch memory-stream-compat
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 3ca4eb6f5577d3f8c67ad0d19a9cff9a6a8e1e5f
Author: Wenchen Fan <we...@databricks.com>
AuthorDate: Tue Feb 3 18:54:00 2026 +0800
[SPARK-53656][SS][FOLLOWUP] Remove confusing MemoryStream factory method
Remove the `apply[A: Encoder](numPartitions: Int, sparkSession: SparkSession)` factory method that creates a semantic trap - it can accidentally match calls like `MemoryStream[T](0, spark)` interpreting the first argument as `numPartitions` instead of `id`, causing zero partitions to be created and no data to flow.
Users who need both `numPartitions` and explicit `SparkSession` can use the case class constructor directly: `new MemoryStream[A](id, sparkSession, Some(numPartitions))`.
Co-authored-by: Cursor <cu...@cursor.com>
---
.../org/apache/spark/sql/execution/streaming/runtime/memory.scala | 7 -------
1 file changed, 7 deletions(-)
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/memory.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/memory.scala
index 306fd45db2bb..e4487b03bd41 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/memory.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/memory.scala
@@ -78,13 +78,6 @@ object MemoryStream {
*/
def apply[A](encoder: Encoder[A], sparkSession: SparkSession): MemoryStream[A] =
new MemoryStream[A](memoryStreamId.getAndIncrement(), sparkSession)(encoder)
-
- /**
- * Creates a MemoryStream with specified partitions using explicit SparkSession.
- * Usage: `MemoryStream[Int](numPartitions, spark)`
- */
- def apply[A: Encoder](numPartitions: Int, sparkSession: SparkSession): MemoryStream[A] =
- new MemoryStream[A](memoryStreamId.getAndIncrement(), sparkSession, Some(numPartitions))
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org