You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zifeif2 (via GitHub)" <gi...@apache.org> on 2026/01/20 21:55:27 UTC

[PR] [SPARK-55105][SS] Add Integration Test for Join Operator [spark]

zifeif2 opened a new pull request, #53874:
URL: https://github.com/apache/spark/pull/53874

   
   ### What changes were proposed in this pull request?
   Adding integration test for join operator, including V1, V2 and V3.
   V1 and V2 join operator use multiple state stores with single column family, while V3 uses a single state store with multiple column families
   
   ### Why are the changes needed?
   Adding integration test for all operators to ensure repartition correctness
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
   
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   No
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   Add added test
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   yes. Sonnet 4.5
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55105][SS] Add Integration Test for Join Operator [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53874:
URL: https://github.com/apache/spark/pull/53874#issuecomment-3775131403

   ## JIRA Issue Information
   
   === New Feature SPARK-55105 ===
   Summary: Test repartition for Join
   Assignee: None
   Status: Open
   Affected: ["4.1.1"]
   
   ---
   *This comment was automatically generated by GitHub Actions*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55105][SS] Add Integration Test for Join Operator [spark]

Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #53874:
URL: https://github.com/apache/spark/pull/53874#discussion_r2738884426


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionJoinIntegrationSuite.scala:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.streaming.state
+
+import org.apache.spark.sql.execution.datasources.v2.state.{StateSourceOptions, StreamStreamJoinTestUtils}
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Integration test suite for stream-stream join operator repartitioning.
+ */
+class OfflineStateRepartitionJoinIntegrationSuite
+  extends OfflineStateRepartitionIntegrationSuiteBase {

Review Comment:
   Do we need to add chkptv2 suite for this too. Since given the base it is extending, it will only run for v1 right now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55105][SS] Add Integration Test for Join Operator [spark]

Posted by "zifeif2 (via GitHub)" <gi...@apache.org>.
zifeif2 commented on code in PR #53874:
URL: https://github.com/apache/spark/pull/53874#discussion_r2739114069


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionJoinIntegrationSuite.scala:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.streaming.state
+
+import org.apache.spark.sql.execution.datasources.v2.state.{StateSourceOptions, StreamStreamJoinTestUtils}
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Integration test suite for stream-stream join operator repartitioning.
+ */
+class OfflineStateRepartitionJoinIntegrationSuite
+  extends OfflineStateRepartitionIntegrationSuiteBase {

Review Comment:
   Just added. Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55105][SS] Add Integration Test for Join Operator [spark]

Posted by "zifeif2 (via GitHub)" <gi...@apache.org>.
zifeif2 commented on code in PR #53874:
URL: https://github.com/apache/spark/pull/53874#discussion_r2729576576


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationSuite.scala:
##########
@@ -479,4 +525,47 @@ class OfflineStateRepartitionIntegrationSuite
       }
     )
   }
+
+  Seq(1, 2, 3).foreach { version =>

Review Comment:
   Ok will split it. 
   Just documenting: without join tests. the suite takes 5 minutes, 12 seconds to finish. With join tests, the suite takes  9 minutes, 15 seconds.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55105][SS] Add Integration Test for Join Operator [spark]

Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #53874:
URL: https://github.com/apache/spark/pull/53874#discussion_r2729846095


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationSuite.scala:
##########
@@ -187,7 +231,13 @@ class OfflineStateRepartitionIntegrationSuite
       }
     }
   }
+}
 
+/**
+ * Integration test suite for OfflineStateRepartitionRunner with single-column family operators.
+ */
+class OfflineStateRepartitionIntegrationSuite extends OfflineStateRepartitionIntegrationSuiteBase {
+  import testImplicits._

Review Comment:
   add new line between



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationJoinSuite.scala:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.streaming.state
+
+import org.apache.spark.sql.execution.datasources.v2.state.{StateSourceOptions, StreamStreamJoinTestUtils}
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Integration test suite for stream-stream join operator repartitioning.
+ */
+class OfflineStateRepartitionIntegrationJoinSuite

Review Comment:
   fix class name and file name: `OfflineStateRepartitionJoinIntegrationSuite`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55105][SS] Add Integration Test for Join Operator [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #53874:
URL: https://github.com/apache/spark/pull/53874#discussion_r2748633255


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationSuite.scala:
##########
@@ -134,23 +152,49 @@ class OfflineStateRepartitionIntegrationSuite
       val repartitionBatchId = lastBatchId + 1
       val hadoopConf = spark.sessionState.newHadoopConf()
 
-      // Step 4: Verify offset and commit logs are updated correctly
-      verifyRepartitionBatch(
+      // Step 4: Verify offset and commit logs
+      OfflineStateRepartitionTestUtils.verifyRepartitionBatch(
         repartitionBatchId, checkpointMetadata, hadoopConf,
         checkpointDir.getAbsolutePath, newPartitions, spark)
 
-      // Step 5: Validate state data matches after repartition
-      validateStateDataAfterRepartition(
-        checkpointDir.getAbsolutePath, stateBeforeRepartition.collect(),
-        repartitionBatchId)
+      // Step 5: Validate state for each store and column family after repartition
+      val stateAfterRepartition = readStateDataByStoreName(
+        checkpointDir.getAbsolutePath, repartitionBatchId, storeToColumnFamilyToStateSourceOptions)
+
+      storeToColumnFamilyToStateSourceOptions.foreach { case (storeName, columnFamilies) =>
+        columnFamilies.keys.foreach { cfName =>
+          val beforeState = stateBeforeRepartition(storeName)(cfName)
+          val afterState = stateAfterRepartition(storeName)(cfName)
+
+          // Validate row count
+          assert(beforeState.length == afterState.length,
+            s"Store '$storeName', CF '$cfName': State row count mismatch: " +
+              s"before=${beforeState.length}, after=${afterState.length}")
+
+          // Extract (key, value) pairs and compare
+          val beforeByKey = extractKeyValuePairs(beforeState)
+          val afterByKey = extractKeyValuePairs(afterState)
+
+          // Compare each (key, value) pair
+          beforeByKey.zip(afterByKey).zipWithIndex.foreach {
+            case (((keyBefore, valueBefore), (keyAfter, valueAfter)), idx) =>
+              assert(keyBefore == keyAfter,
+                s"Store '$storeName', CF '$cfName': Key mismatch at index $idx")
+              assert(valueBefore == valueAfter,
+                s"Store '$storeName', CF '$cfName': Value mismatch for key $keyBefore")
+          }
+        }
+      }
 
       // Step 6: Resume query with new input and verify
       verifyResumedQuery(inputData, checkpointDir.getAbsolutePath, clock)
     }
   }
 
   def testWithChangelogConfig(testName: String)(testFun: => Unit): Unit = {
-    Seq(true, false).foreach { changelogCheckpointingEnabled =>
+    // TODO: add test with changelog checkpointing disabled after SPARK increases its test timeout

Review Comment:
   Can we create a SPARK JIRA for this and tag it here @zifeif2 ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55105][SS] Add Integration Test for Join Operator [spark]

Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #53874:
URL: https://github.com/apache/spark/pull/53874#discussion_r2729354376


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationSuite.scala:
##########
@@ -479,4 +525,47 @@ class OfflineStateRepartitionIntegrationSuite
       }
     )
   }
+
+  Seq(1, 2, 3).foreach { version =>

Review Comment:
   Lets probably make join test case a separate suite e.g. `OfflineStateRepartitionJoinIntegrationSuite`



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationSuite.scala:
##########
@@ -479,4 +525,47 @@ class OfflineStateRepartitionIntegrationSuite
       }
     )
   }
+
+  Seq(1, 2, 3).foreach { version =>
+    testWithAllRepartitionOperations(s"stream-stream join ver $version") { newPartitions =>
+      withSQLConf(SQLConf.STREAMING_JOIN_STATE_FORMAT_VERSION.key -> version.toString) {
+        val allStoreNames = StreamStreamJoinTestUtils.allStoreNames
+        val storeToOptions = if (version <= 2) {
+          allStoreNames.map { storeName =>
+            storeName -> Map(StateStore.DEFAULT_COL_FAMILY_NAME -> Map.empty[String, String])
+          }.toMap
+        } else {
+          Map(StateStoreId.DEFAULT_STORE_NAME -> allStoreNames.map { colFamilyName =>
+            colFamilyName -> Map(StateSourceOptions.STORE_NAME -> colFamilyName)
+          }.toMap)
+        }
+
+        testRepartitionWorkflow[(Int, Long)](
+          newPartitions = newPartitions,
+          setupInitialState = (inputData, checkpointDir, _) => {
+            val query = getStreamStreamJoinQuery(inputData)
+            testStream(query)(
+              StartStream(checkpointLocation = checkpointDir),
+              // Batch 1: Creates state in all 4 column families
+              AddData(inputData, (1, 1L), (2, 2L), (3, 3L), (4, 4L), (5, 5L)),
+              CheckNewAnswer((2, 2, 2, 2), (4, 4, 4, 4)),
+              // Batch 2: Adds more state to all column families
+              AddData(inputData, (6, 6L), (7, 7L), (8, 8L), (9, 9L), (10, 10L)),
+              CheckNewAnswer((6, 6, 6, 6), (8, 8, 8, 8), (10, 10, 10, 10)),
+              StopStream
+            )
+          },
+          verifyResumedQuery = (inputData, checkpointDir, _) => {
+            val query = getStreamStreamJoinQuery(inputData)
+            testStream(query)(
+              StartStream(checkpointLocation = checkpointDir),
+              AddData(inputData, (6, 10L)),

Review Comment:
   nit: can add 1 or 2 more data here. Just to make sure other existing rows match too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55105][SS] Add Integration Test for Join Operator [spark]

Posted by "zifeif2 (via GitHub)" <gi...@apache.org>.
zifeif2 commented on code in PR #53874:
URL: https://github.com/apache/spark/pull/53874#discussion_r2733553149


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/OfflineStateRepartitionIntegrationSuite.scala:
##########
@@ -25,17 +25,13 @@ import org.apache.spark.sql.streaming.{OutputMode, Trigger}
 import org.apache.spark.sql.streaming.util.StreamManualClock
 
 /**
- * Integration test suite for OfflineStateRepartitionRunner with stateful operators.
+ * Integration test suite helper class for OfflineStateRepartitionRunner with stateful operators.
  * Tests state repartitioning (increase and decrease partitions) and validates:
  * 1. State data remains identical after repartitioning
  * 2. Offset and commit logs are updated correctly
  * 3. Query can resume successfully and compute correctly with repartitioned state
  */
-class OfflineStateRepartitionIntegrationSuite
-  extends StateDataSourceTestBase {
-
-  import testImplicits._
-  import OfflineStateRepartitionTestUtils._
+abstract class OfflineStateRepartitionIntegrationSuiteBase extends StateDataSourceTestBase {

Review Comment:
   Changing this to abstract class instead of trait because here is the inheritance chain
   ```
     1. OfflineStateRepartitionIntegrationSuite (class) extends OfflineStateRepartitionIntegrationSuiteBase (trait)                                                                                                                         
     2. OfflineStateRepartitionIntegrationSuiteBase (trait) extends StateDataSourceTestBase (trait)                                                                                                                                         
     3. StateDataSourceTestBase (trait) extends StreamTest (trait)                                                                                                                                                                          
     4. StreamTest (trait) extends QueryTest (abstract class)                                                                                                                                                                               
     5. QueryTest (abstract class) extends PlanTest (trait)                                                                                                                                                                                 
     6. PlanTest (trait) extends SparkFunSuite (class) 
     ```
   which means that there is a ` concrete class → trait → trait → abstract class → trait → class`.
    In Scala, traits cannot safely hide or defer concrete superclasses for concrete subclasses; the concrete superclass must be resolved at a class boundary. Making this an abstract class allows it to legally “anchor” the concrete superclass in the inheritance chain, ensuring correct class linearization and avoiding illegal inheritance errors during compilation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55105][SS] Add Integration Test for Join Operator [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db closed pull request #53874: [SPARK-55105][SS] Add Integration Test for Join Operator
URL: https://github.com/apache/spark/pull/53874


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org