You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zifeif2 (via GitHub)" <gi...@apache.org> on 2025/12/08 17:52:13 UTC

[PR] [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF [spark]

zifeif2 opened a new pull request, #53386:
URL: https://github.com/apache/spark/pull/53386

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'common/utils/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   Introduce Repartition Writer that supports operator with multiple column families. This include
   **TransformWithState**
   - Support List, Map, Value state variables
   - event time timers, processing time timers and TTLs 
   - Stream Stream Join V3
   
   Other things that this PR makes:
   - When reading a LIST TWS state, it's returned as unnested operator (one element per row) [PR](https://github.com/apache/spark/pull/53316). Therefore, when writing a list back to the State Store, writer will use "db.put()" followed by "db.merge()"
   
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   To complete the Offline Repartitioning project
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
   
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   No
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   See integration test added for the operators mentioned above
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF [spark]

Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #53386:
URL: https://github.com/apache/spark/pull/53386#discussion_r2632581091


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -47,30 +52,39 @@ class StatePartitionAllColumnFamiliesWriter(
     operatorId: Int,
     storeName: String,
     currentBatchId: Long,
-    columnFamilyToSchemaMap: HashMap[String, StateStoreColFamilySchema]) {
+    columnFamilyToSchemaMap: HashMap[String, StatePartitionWriterColumnFamilyInfo]) {
   private val defaultSchema = {
-    columnFamilyToSchemaMap.getOrElse(
-      StateStore.DEFAULT_COL_FAMILY_NAME,
-      throw new IllegalArgumentException(
-        s"Column family ${StateStore.DEFAULT_COL_FAMILY_NAME} not found in schema map")
-    )
+    columnFamilyToSchemaMap.get(StateStore.DEFAULT_COL_FAMILY_NAME) match {
+      case Some(info) => info.schema
+      case None =>
+        // Return a dummy StateStoreColFamilySchema if not found
+        val placeholderSchema = columnFamilyToSchemaMap.head._2.schema
+        StateStoreColFamilySchema(
+          colFamilyName = "__dummy__",
+          keySchemaId = 0,
+          keySchema = placeholderSchema.keySchema,
+          valueSchemaId = 0,
+          valueSchema = placeholderSchema.valueSchema,
+          keyStateEncoderSpec = Option(NoPrefixKeyStateEncoderSpec(placeholderSchema.keySchema)))
+    }
   }
 
   private val columnFamilyToKeySchemaLenMap: MapView[String, Int] =
-    columnFamilyToSchemaMap.view.mapValues(_.keySchema.length)
+    columnFamilyToSchemaMap.view.mapValues(_.schema.keySchema.length)
   private val columnFamilyToValueSchemaLenMap: MapView[String, Int] =
-    columnFamilyToSchemaMap.view.mapValues(_.valueSchema.length)
+    columnFamilyToSchemaMap.view.mapValues(_.schema.valueSchema.length)
 
   protected lazy val provider: StateStoreProvider = {
     val stateCheckpointLocation = new Path(targetCpLocation, DIR_NAME_STATE).toString
     val stateStoreId = StateStoreId(stateCheckpointLocation,
       operatorId, partitionId, storeName)
     val stateStoreProviderId = StateStoreProviderId(stateStoreId, UUID.randomUUID())
 
+    val useColumnFamilies = columnFamilyToSchemaMap.size > 1
     val provider = StateStoreProvider.createAndInit(
       stateStoreProviderId, defaultSchema.keySchema, defaultSchema.valueSchema,
       defaultSchema.keyStateEncoderSpec.get,
-      useColumnFamilies = false, storeConf, hadoopConf,
+      useColumnFamilies = useColumnFamilies, storeConf, hadoopConf,
       useMultipleValuesPerKey = false, stateSchemaProvider = None)

Review Comment:
   `stateSchemaProvider` must be passed in, otherwise it won't work with Avro state format.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db closed pull request #53386: [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF 
URL: https://github.com/apache/spark/pull/53386


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF [spark]

Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #53386:
URL: https://github.com/apache/spark/pull/53386#discussion_r2625003265


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -146,18 +148,47 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
     assert(!checkpointFileExists(new File(targetDir, storeNamePath), versionToCheck, ".changelog"))
     assert(checkpointFileExists(new File(targetDir, storeNamePath), versionToCheck, ".zip"))
 
-    // Step 4: Read from target using normal reader
-    val targetReader = spark.read
-      .format("statestore")
-      .option(StateSourceOptions.PATH, targetDir)
-    val targetNormalData = (storeName match {
-      case Some(name) => targetReader.option(StateSourceOptions.STORE_NAME, name)
-      case None => targetReader
-    }).load()
-      .selectExpr("key", "value", "partition_id")
-      .collect()
+    // Step 3: Validate by reading from both source and target using normal reader"
+    // Default selectExprs for most column families
+    val defaultSelectExprs = Seq("key", "value", "partition_id")
+
+    def shouldCheckColumnFamilyName: String => Boolean = name => {
+      (!name.startsWith("$")
+        || (columnFamilyToStateSourceOptions.contains(name) &&

Review Comment:
   please add comment to explain what this func is doing



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -146,18 +148,47 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
     assert(!checkpointFileExists(new File(targetDir, storeNamePath), versionToCheck, ".changelog"))
     assert(checkpointFileExists(new File(targetDir, storeNamePath), versionToCheck, ".zip"))
 
-    // Step 4: Read from target using normal reader
-    val targetReader = spark.read
-      .format("statestore")
-      .option(StateSourceOptions.PATH, targetDir)
-    val targetNormalData = (storeName match {
-      case Some(name) => targetReader.option(StateSourceOptions.STORE_NAME, name)
-      case None => targetReader
-    }).load()
-      .selectExpr("key", "value", "partition_id")
-      .collect()
+    // Step 3: Validate by reading from both source and target using normal reader"
+    // Default selectExprs for most column families
+    val defaultSelectExprs = Seq("key", "value", "partition_id")
+
+    def shouldCheckColumnFamilyName: String => Boolean = name => {
+      (!name.startsWith("$")
+        || (columnFamilyToStateSourceOptions.contains(name) &&
+        columnFamilyToStateSourceOptions(name).contains(StateSourceOptions.READ_REGISTERED_TIMERS)))
+    }
+    // Validate each column family separately (skip internal column families starting with $)

Review Comment:
   lets also handle internal cf



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -47,30 +52,39 @@ class StatePartitionAllColumnFamiliesWriter(
     operatorId: Int,
     storeName: String,
     currentBatchId: Long,
-    columnFamilyToSchemaMap: HashMap[String, StateStoreColFamilySchema]) {
+    columnFamilyToSchemaMap: HashMap[String, StatePartitionWriterColumnFamilyInfo]) {
   private val defaultSchema = {
-    columnFamilyToSchemaMap.getOrElse(
-      StateStore.DEFAULT_COL_FAMILY_NAME,
-      throw new IllegalArgumentException(
-        s"Column family ${StateStore.DEFAULT_COL_FAMILY_NAME} not found in schema map")
-    )
+    columnFamilyToSchemaMap.get(StateStore.DEFAULT_COL_FAMILY_NAME) match {
+      case Some(info) => info.schema
+      case None =>
+        // Return a dummy StateStoreColFamilySchema if not found
+        val placeholderSchema = columnFamilyToSchemaMap.head._2.schema

Review Comment:
   We are doing this just for joinv3 right? Then lets do it for only it. And throw for other cases



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -82,11 +96,33 @@ class StatePartitionAllColumnFamiliesWriter(
     // Use loadEmpty=true to create a fresh state store without loading previous versions
     // We create the empty store AT version, and the next commit will
     // produce version + 1
-    provider.getStore(
+    val store = provider.getStore(
       currentBatchId,
       stateStoreCkptId = None,
       loadEmpty = true
     )
+    if (columnFamilyToSchemaMap.size > 1) {

Review Comment:
   nit: can just have a class val `useColumnFamilies` and you can use it here and the other place



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -468,15 +593,373 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
           performRoundTripTest(
             sourceDir.getAbsolutePath,
             targetDir.getAbsolutePath,
-            keySchema,
-            valueSchema,
-            keyStateEncoderSpec
+            createSingleColumnFamilySchemaMap(keySchema, valueSchema, keyStateEncoderSpec)
           )
         }
       }
     }
   }
 
+  /**
+   * Helper method to test round-trip for transformWithState with multiple column families.
+   * Uses MultiStateVarProcessor which creates ValueState, ListState, and MapState.
+   */
+  private def testTransformWithStateMultiColumnFamilies(): Unit = {
+    withTempDir { sourceDir =>
+      withTempDir { targetDir =>
+        val inputData = MemoryStream[String]
+        val query = inputData.toDS()
+          .groupByKey(x => x)
+          .transformWithState(new MultiStateVarProcessor(),
+            TimeMode.None(),
+            OutputMode.Update())
+        def runQuery(checkpointLocation: String, roundsOfData: Int): Unit = {
+          val dataActions = (1 to roundsOfData).flatMap { _ =>
+            Seq(
+              AddData(inputData, "a", "b", "a"),
+              ProcessAllAvailable()
+            )
+          }
+          testStream(query)(
+            Seq(StartStream(checkpointLocation = checkpointLocation)) ++
+              dataActions ++
+              Seq(StopStream): _*
+          )
+        }
+
+        // Step 1: Add data to source
+        runQuery(sourceDir.getAbsolutePath, 2)
+        // Step 2: Add data to target
+        runQuery(targetDir.getAbsolutePath, 1)
+
+        // Step 3: Define schemas for all column families
+        val groupByKeySchema = StructType(Array(

Review Comment:
   ditto, common test util?



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -82,11 +96,33 @@ class StatePartitionAllColumnFamiliesWriter(
     // Use loadEmpty=true to create a fresh state store without loading previous versions
     // We create the empty store AT version, and the next commit will
     // produce version + 1
-    provider.getStore(
+    val store = provider.getStore(
       currentBatchId,
       stateStoreCkptId = None,
       loadEmpty = true
     )
+    if (columnFamilyToSchemaMap.size > 1) {
+      columnFamilyToSchemaMap.foreach { pair =>
+        val colFamilyName = pair._1
+        val cfSchema = pair._2.schema
+        colFamilyName match {
+          case StateStore.DEFAULT_COL_FAMILY_NAME => // createAndInit has registered default
+          case _ =>
+            val isInternal = StateStoreColumnFamilySchemaUtils.isInternalColFamily(colFamilyName)
+
+            require(cfSchema.keyStateEncoderSpec.isDefined,
+              s"keyStateEncoderSpec must be defined for column family ${cfSchema.colFamilyName}")
+            store.createColFamilyIfAbsent(
+              colFamilyName,
+              cfSchema.keySchema,
+              cfSchema.valueSchema,
+              cfSchema.keyStateEncoderSpec.get,
+              columnFamilyToSchemaMap(colFamilyName).useMultipleValuesPerKey,

Review Comment:
   you already have this: `pair._2.useMultipleValuesPerKey` right?



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -676,5 +1151,25 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
         testStreamStreamJoinRoundTrip(version)
       }
     }
+
+    testWithChangelogConfig("SPARK-54411: stream-stream join state ver 3") {

Review Comment:
   for funcs that are only used ones, no need to create separate func for them. Just implement it within the test case. We use func, when the func will be called by multiple test cases



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -123,6 +159,12 @@ class StatePartitionAllColumnFamiliesWriter(
     val valueRow = new UnsafeRow(columnFamilyToValueSchemaLenMap(colFamilyName))
     valueRow.pointTo(valueBytes, valueBytes.length)
 
-    stateStore.put(keyRow, valueRow, colFamilyName)
+    if (columnFamilyToSchemaMap(colFamilyName).useMultipleValuesPerKey) {
+      // if a column family useMultipleValuesPerKey (e.g. ListType), we will
+      // write with 1 put followed by merge
+      stateStore.merge(keyRow, valueRow, colFamilyName)

Review Comment:
   This is wrong right. You said it in your comment here, to do 1 put and then merge for the rest. But you are only doing merge for everything. You can use `stateStore.keyExists(keyRow, colFamilyName)` to determine whether to put or merge.



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -130,9 +118,23 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
         currentBatchId,
         columnFamilyToSchemaMap
       )
-      val rowConverter = CatalystTypeConverters.createToCatalystConverter(schema)
 
-      allCFWriter.write(partition.map(rowConverter(_).asInstanceOf[InternalRow]))
+      // Use per-column-family converters when there are multiple column families
+      if (columnFamilyToSchemaMap.size > 1) {
+        // TODO: Remove the logic of getting colNameToRowConverter once allColumnFamiliesReader is
+        // returning actual partitionKeySchema instead of the entire key
+        val colNameToRowConverter = columnFamilyToSchemaMap.view.mapValues { colInfo =>

Review Comment:
   nit: cfNameToRowConverter instead?



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -203,6 +234,37 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
         }
     }
 
+  private def createColFamilyInfo(
+       keySchema: StructType,
+       valueSchema: StructType,
+       keyStateEncoderSpec: KeyStateEncoderSpec,
+       colFamilyName: String,
+       useMultipleValuePerKey: Boolean = false): StatePartitionWriterColumnFamilyInfo = {
+    StatePartitionWriterColumnFamilyInfo(
+      schema = StateStoreColFamilySchema(
+        colFamilyName,
+        keySchemaId = 0,
+        keySchema,
+        valueSchemaId = 0,
+        valueSchema,
+        keyStateEncoderSpec = Some(keyStateEncoderSpec)
+      ),
+      useMultipleValuePerKey)
+  }

Review Comment:
   new line



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -329,15 +389,42 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
           performRoundTripTest(
             sourceDir.getAbsolutePath,
             targetDir.getAbsolutePath,
-            keySchema,
-            valueSchema,
-            keyStateEncoderSpec
+            createSingleColumnFamilySchemaMap(keySchema, valueSchema, keyStateEncoderSpec)
           )
         }
       }
     }
   }
 
+  private val keyToNumValuesColFamilyNames = Seq("left-keyToNumValues", "right-keyToNumValues")

Review Comment:
   Many of these are duplicate of what we do in the multi-cf reader suite too. Should we move them to a common test util? Or are they different?



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -53,31 +57,27 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
    *
    * @param sourceDir Source checkpoint directory
    * @param targetDir Target checkpoint directory
-   * @param keySchema Key schema for the state store
-   * @param valueSchema Value schema for the state store
-   * @param keyStateEncoderSpec Key state encoder spec
+   * @param columnFamilyToSchemaMap Map of column family names to their schemas
    * @param storeName Optional store name (for stream-stream join which has multiple stores)
+   * @param columnFamilyToSelectExprs Map of column family names to custom selectExprs
+   * @param columnFamilyToStateSourceOptions Map of column family names to state source options
    */
   private def performRoundTripTest(
       sourceDir: String,
       targetDir: String,
-      keySchema: StructType,
-      valueSchema: StructType,
-      keyStateEncoderSpec: KeyStateEncoderSpec,
-      storeName: Option[String] = None): Unit = {
-
-    // Step 1: Read original state using normal reader (for comparison later)
-    val sourceReader = spark.read
-      .format("statestore")
-      .option(StateSourceOptions.PATH, sourceDir)
-    val sourceNormalData = (storeName match {
-      case Some(name) => sourceReader.option(StateSourceOptions.STORE_NAME, name)
-      case None => sourceReader
-    }).load()
-      .selectExpr("key", "value", "partition_id")
-      .collect()
+      columnFamilyToSchemaMap: HashMap[String, StatePartitionWriterColumnFamilyInfo],
+      storeName: Option[String] = None,
+      columnFamilyToSelectExprs: Map[String, Seq[String]] = Map.empty,
+      columnFamilyToStateSourceOptions: Map[String, Map[String, String]] = Map.empty): Unit = {
+
+    // Determine column families to validate based on storeName and map size
+    val columnFamiliesToValidate: Seq[String] = storeName match {
+      case Some(name) => Seq(name)

Review Comment:
   why?



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -468,15 +593,373 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
           performRoundTripTest(
             sourceDir.getAbsolutePath,
             targetDir.getAbsolutePath,
-            keySchema,
-            valueSchema,
-            keyStateEncoderSpec
+            createSingleColumnFamilySchemaMap(keySchema, valueSchema, keyStateEncoderSpec)
           )
         }
       }
     }
   }
 
+  /**
+   * Helper method to test round-trip for transformWithState with multiple column families.
+   * Uses MultiStateVarProcessor which creates ValueState, ListState, and MapState.
+   */
+  private def testTransformWithStateMultiColumnFamilies(): Unit = {
+    withTempDir { sourceDir =>
+      withTempDir { targetDir =>
+        val inputData = MemoryStream[String]
+        val query = inputData.toDS()
+          .groupByKey(x => x)
+          .transformWithState(new MultiStateVarProcessor(),
+            TimeMode.None(),
+            OutputMode.Update())
+        def runQuery(checkpointLocation: String, roundsOfData: Int): Unit = {
+          val dataActions = (1 to roundsOfData).flatMap { _ =>
+            Seq(
+              AddData(inputData, "a", "b", "a"),
+              ProcessAllAvailable()
+            )
+          }
+          testStream(query)(
+            Seq(StartStream(checkpointLocation = checkpointLocation)) ++
+              dataActions ++
+              Seq(StopStream): _*
+          )
+        }
+
+        // Step 1: Add data to source
+        runQuery(sourceDir.getAbsolutePath, 2)
+        // Step 2: Add data to target
+        runQuery(targetDir.getAbsolutePath, 1)
+
+        // Step 3: Define schemas for all column families
+        val groupByKeySchema = StructType(Array(
+          StructField("value", StringType, nullable = true)
+        ))
+        val countStateValueSchema = StructType(Array(
+          StructField("value", LongType, nullable = false)
+        ))
+        val itemsListValueSchema = StructType(Array(
+          StructField("value", StringType, nullable = true)
+        ))
+        val rowCounterValueSchema = StructType(Array(
+          StructField("count", LongType, nullable = true)
+        ))
+        val itemsMapKeySchema = StructType(Array(
+          StructField("key", StringType),
+          StructField("user_map_key", groupByKeySchema, nullable = true)
+        ))
+        val itemsMapValueSchema = StructType(Array(
+          StructField("user_map_value", IntegerType, nullable = true)
+        ))
+
+        // Build column family to schema map for all 4 state variables
+        val countStateEncoderSpec = NoPrefixKeyStateEncoderSpec(groupByKeySchema)
+        val itemsMapEncoderSpec = PrefixKeyScanStateEncoderSpec(itemsMapKeySchema, 1)
+
+        val columnFamilyToSchemaMap = HashMap(
+          "countState" -> createColFamilyInfo(
+            groupByKeySchema, countStateValueSchema, countStateEncoderSpec, "countState"),
+          "itemsList" -> createColFamilyInfo(
+            groupByKeySchema, itemsListValueSchema, countStateEncoderSpec, "itemsList", true),
+          "$rowCounter_itemsList" -> createColFamilyInfo(
+            groupByKeySchema, rowCounterValueSchema,
+            countStateEncoderSpec, "$rowCounter_itemsList"),
+          "itemsMap" -> createColFamilyInfo(
+            itemsMapKeySchema, itemsMapValueSchema, itemsMapEncoderSpec, "itemsMap")
+        )
+
+        // Define custom selectExprs for column families with non-standard schemas
+        val columnFamilyToSelectExprs = Map(
+          "itemsList" -> Seq("key", "list_element AS value", "partition_id"),
+          "itemsMap" -> Seq("STRUCT(key, user_map_key) AS key", "user_map_value AS value",
+            "partition_id")
+        )
+
+        // Define reader options for column families that need them
+        val columnFamilyToStateSourceOptions = Map(
+          "itemsList" -> Map(StateSourceOptions.FLATTEN_COLLECTION_TYPES -> "true",
+            StateSourceOptions.STATE_VAR_NAME -> "itemsList"),
+          "itemsMap" -> Map(StateSourceOptions.STATE_VAR_NAME -> "itemsMap"),
+          "countState" -> Map(StateSourceOptions.STATE_VAR_NAME -> "countState")
+        )
+
+        // Perform round-trip test using common helper
+        performRoundTripTest(
+          sourceDir.getAbsolutePath,
+          targetDir.getAbsolutePath,
+          columnFamilyToSchemaMap,
+          columnFamilyToSelectExprs = columnFamilyToSelectExprs,
+          columnFamilyToStateSourceOptions = columnFamilyToStateSourceOptions
+        )
+      }
+    }
+  }
+
+  /**
+   * Helper method to build timer column family schemas and options for timerProcesser
+   * that has groupingKey of STRING type and keeps track of a "countState" of LONG type.
+   * Used by both event time and processing time timer tests
+   *
+   * @param timeMode Either TimeMode.EventTime() or TimeMode.ProcessingTime()
+   * @return A tuple of three elements:
+   *         - columnFamilyToSchemaMap: Maps column family names to their schema info
+   *         - columnFamilyToSelectExprs: Maps column family names to custom select expressions
+   *         - columnFamilyToStateSourceOptions: Maps column family names to state source options
+   */
+  private def getTimerStateConfigsForCountState(timeMode: TimeMode): (
+      HashMap[String, StatePartitionWriterColumnFamilyInfo],
+      Map[String, Seq[String]],
+      Map[String, Map[String, String]]) = {
+
+    val groupByKeySchema = StructType(Array(

Review Comment:
   ditto, common test util?



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -53,31 +57,27 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
    *
    * @param sourceDir Source checkpoint directory
    * @param targetDir Target checkpoint directory
-   * @param keySchema Key schema for the state store
-   * @param valueSchema Value schema for the state store
-   * @param keyStateEncoderSpec Key state encoder spec
+   * @param columnFamilyToSchemaMap Map of column family names to their schemas
    * @param storeName Optional store name (for stream-stream join which has multiple stores)
+   * @param columnFamilyToSelectExprs Map of column family names to custom selectExprs
+   * @param columnFamilyToStateSourceOptions Map of column family names to state source options
    */
   private def performRoundTripTest(
       sourceDir: String,
       targetDir: String,
-      keySchema: StructType,
-      valueSchema: StructType,
-      keyStateEncoderSpec: KeyStateEncoderSpec,
-      storeName: Option[String] = None): Unit = {
-
-    // Step 1: Read original state using normal reader (for comparison later)
-    val sourceReader = spark.read
-      .format("statestore")
-      .option(StateSourceOptions.PATH, sourceDir)
-    val sourceNormalData = (storeName match {
-      case Some(name) => sourceReader.option(StateSourceOptions.STORE_NAME, name)
-      case None => sourceReader
-    }).load()
-      .selectExpr("key", "value", "partition_id")
-      .collect()
+      columnFamilyToSchemaMap: HashMap[String, StatePartitionWriterColumnFamilyInfo],
+      storeName: Option[String] = None,
+      columnFamilyToSelectExprs: Map[String, Seq[String]] = Map.empty,
+      columnFamilyToStateSourceOptions: Map[String, Map[String, String]] = Map.empty): Unit = {
+
+    // Determine column families to validate based on storeName and map size
+    val columnFamiliesToValidate: Seq[String] = storeName match {
+      case Some(name) => Seq(name)
+      case None if columnFamilyToSchemaMap.size > 1 => columnFamilyToSchemaMap.keys.toSeq
+      case None => Seq(StateStoreId.DEFAULT_STORE_NAME)

Review Comment:
   nit: DEFAULT_COL_FAMILY instead?



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -468,15 +593,373 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
           performRoundTripTest(
             sourceDir.getAbsolutePath,
             targetDir.getAbsolutePath,
-            keySchema,
-            valueSchema,
-            keyStateEncoderSpec
+            createSingleColumnFamilySchemaMap(keySchema, valueSchema, keyStateEncoderSpec)
           )
         }
       }
     }
   }
 
+  /**
+   * Helper method to test round-trip for transformWithState with multiple column families.
+   * Uses MultiStateVarProcessor which creates ValueState, ListState, and MapState.
+   */
+  private def testTransformWithStateMultiColumnFamilies(): Unit = {
+    withTempDir { sourceDir =>
+      withTempDir { targetDir =>
+        val inputData = MemoryStream[String]
+        val query = inputData.toDS()
+          .groupByKey(x => x)
+          .transformWithState(new MultiStateVarProcessor(),
+            TimeMode.None(),
+            OutputMode.Update())
+        def runQuery(checkpointLocation: String, roundsOfData: Int): Unit = {
+          val dataActions = (1 to roundsOfData).flatMap { _ =>
+            Seq(
+              AddData(inputData, "a", "b", "a"),
+              ProcessAllAvailable()
+            )
+          }
+          testStream(query)(
+            Seq(StartStream(checkpointLocation = checkpointLocation)) ++
+              dataActions ++
+              Seq(StopStream): _*
+          )
+        }
+
+        // Step 1: Add data to source
+        runQuery(sourceDir.getAbsolutePath, 2)
+        // Step 2: Add data to target
+        runQuery(targetDir.getAbsolutePath, 1)
+
+        // Step 3: Define schemas for all column families
+        val groupByKeySchema = StructType(Array(
+          StructField("value", StringType, nullable = true)
+        ))
+        val countStateValueSchema = StructType(Array(
+          StructField("value", LongType, nullable = false)
+        ))
+        val itemsListValueSchema = StructType(Array(
+          StructField("value", StringType, nullable = true)
+        ))
+        val rowCounterValueSchema = StructType(Array(
+          StructField("count", LongType, nullable = true)
+        ))
+        val itemsMapKeySchema = StructType(Array(
+          StructField("key", StringType),
+          StructField("user_map_key", groupByKeySchema, nullable = true)
+        ))
+        val itemsMapValueSchema = StructType(Array(
+          StructField("user_map_value", IntegerType, nullable = true)
+        ))
+
+        // Build column family to schema map for all 4 state variables
+        val countStateEncoderSpec = NoPrefixKeyStateEncoderSpec(groupByKeySchema)
+        val itemsMapEncoderSpec = PrefixKeyScanStateEncoderSpec(itemsMapKeySchema, 1)
+
+        val columnFamilyToSchemaMap = HashMap(
+          "countState" -> createColFamilyInfo(
+            groupByKeySchema, countStateValueSchema, countStateEncoderSpec, "countState"),
+          "itemsList" -> createColFamilyInfo(
+            groupByKeySchema, itemsListValueSchema, countStateEncoderSpec, "itemsList", true),
+          "$rowCounter_itemsList" -> createColFamilyInfo(
+            groupByKeySchema, rowCounterValueSchema,
+            countStateEncoderSpec, "$rowCounter_itemsList"),
+          "itemsMap" -> createColFamilyInfo(
+            itemsMapKeySchema, itemsMapValueSchema, itemsMapEncoderSpec, "itemsMap")
+        )
+
+        // Define custom selectExprs for column families with non-standard schemas
+        val columnFamilyToSelectExprs = Map(
+          "itemsList" -> Seq("key", "list_element AS value", "partition_id"),
+          "itemsMap" -> Seq("STRUCT(key, user_map_key) AS key", "user_map_value AS value",
+            "partition_id")
+        )
+
+        // Define reader options for column families that need them
+        val columnFamilyToStateSourceOptions = Map(
+          "itemsList" -> Map(StateSourceOptions.FLATTEN_COLLECTION_TYPES -> "true",
+            StateSourceOptions.STATE_VAR_NAME -> "itemsList"),
+          "itemsMap" -> Map(StateSourceOptions.STATE_VAR_NAME -> "itemsMap"),
+          "countState" -> Map(StateSourceOptions.STATE_VAR_NAME -> "countState")
+        )
+
+        // Perform round-trip test using common helper
+        performRoundTripTest(
+          sourceDir.getAbsolutePath,
+          targetDir.getAbsolutePath,
+          columnFamilyToSchemaMap,
+          columnFamilyToSelectExprs = columnFamilyToSelectExprs,
+          columnFamilyToStateSourceOptions = columnFamilyToStateSourceOptions
+        )
+      }
+    }
+  }
+
+  /**
+   * Helper method to build timer column family schemas and options for timerProcesser
+   * that has groupingKey of STRING type and keeps track of a "countState" of LONG type.
+   * Used by both event time and processing time timer tests
+   *
+   * @param timeMode Either TimeMode.EventTime() or TimeMode.ProcessingTime()
+   * @return A tuple of three elements:
+   *         - columnFamilyToSchemaMap: Maps column family names to their schema info
+   *         - columnFamilyToSelectExprs: Maps column family names to custom select expressions
+   *         - columnFamilyToStateSourceOptions: Maps column family names to state source options
+   */
+  private def getTimerStateConfigsForCountState(timeMode: TimeMode): (
+      HashMap[String, StatePartitionWriterColumnFamilyInfo],
+      Map[String, Seq[String]],
+      Map[String, Map[String, String]]) = {
+
+    val groupByKeySchema = StructType(Array(
+      StructField("key", StringType, nullable = true)
+    ))
+    val stateValueSchema = StructType(Array(
+      StructField("value", LongType, nullable = true)
+    ))
+    val keyToTimestampKeySchema = StructType(Array(
+      StructField("key", StringType),
+      StructField("expiryTimestampMs", LongType, nullable = false)
+    ))
+    val timestampToKeyKeySchema = StructType(Array(
+      StructField("expiryTimestampMs", LongType, nullable = false),
+      StructField("key", StringType)
+    ))
+    val dummyValueSchema = StructType(Array(StructField("__dummy__", NullType)))
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(groupByKeySchema)
+    val keyToTimestampEncoderSpec = PrefixKeyScanStateEncoderSpec(keyToTimestampKeySchema, 1)
+    val timestampToKeyEncoderSpec = RangeKeyScanStateEncoderSpec(timestampToKeyKeySchema, Seq(0))
+
+    val (keyToTimestampCF, timestampToKeyCF) =
+      TimerStateUtils.getTimerStateVarNames(timeMode.toString)
+
+    val columnFamilyToSchemaMap = HashMap(
+      "countState" -> createColFamilyInfo(
+        groupByKeySchema, stateValueSchema, encoderSpec, "countState"),
+      keyToTimestampCF -> createColFamilyInfo(
+        keyToTimestampKeySchema, dummyValueSchema, keyToTimestampEncoderSpec, keyToTimestampCF),
+      timestampToKeyCF -> createColFamilyInfo(
+        timestampToKeyKeySchema, dummyValueSchema, timestampToKeyEncoderSpec, timestampToKeyCF)
+    )
+
+    val columnFamilyToSelectExprs = Map(
+      keyToTimestampCF -> Seq(
+        "STRUCT(key AS groupingKey, expiration_timestamp_ms AS key) AS key",
+        "NULL AS value", "partition_id"),
+      timestampToKeyCF -> Seq(
+        "STRUCT(expiration_timestamp_ms AS key, key AS groupingKey) AS key",
+        "NULL AS value", "partition_id")
+    )
+
+    val columnFamilyToStateSourceOptions = Map(
+      "countState" -> Map(StateSourceOptions.STATE_VAR_NAME -> "countState"),
+      keyToTimestampCF -> Map(StateSourceOptions.READ_REGISTERED_TIMERS -> "true"),
+      timestampToKeyCF -> Map(StateSourceOptions.READ_REGISTERED_TIMERS -> "true")
+    )
+
+    (columnFamilyToSchemaMap, columnFamilyToSelectExprs, columnFamilyToStateSourceOptions)
+  }
+
+  /**
+   * Helper method to test round-trip for transformWithState with event time timers.
+   */
+  private def testEventTimeTimersRoundTrip(): Unit = {
+    withTempDir { sourceDir =>
+      withTempDir { targetDir =>
+        val inputData = MemoryStream[(String, Long)]
+        val result = inputData.toDS()
+          .select(col("_1").as("key"), timestamp_seconds(col("_2")).as("eventTime"))
+          .withWatermark("eventTime", "10 seconds")
+          .as[(String, Timestamp)]
+          .groupByKey(_._1)
+          .transformWithState(
+            new EventTimeTimerProcessor(),
+            TimeMode.EventTime(),
+            OutputMode.Update())
+
+        // Step 1: Create source checkpoint
+        testStream(result, OutputMode.Update())(
+          StartStream(checkpointLocation = sourceDir.getAbsolutePath),
+          AddData(inputData, ("a", 1L), ("b", 2L), ("c", 3L)),
+          ProcessAllAvailable(),
+          StopStream
+        )
+
+        // Step 2: Create target checkpoint with dummy data
+        testStream(result, OutputMode.Update())(
+          StartStream(checkpointLocation = targetDir.getAbsolutePath),
+          AddData(inputData, ("x", 1L)),
+          ProcessAllAvailable(),
+          StopStream
+        )
+
+        // Step 3: Build timer column family configs and perform round-trip test
+        val (columnFamilyToSchemaMap, columnFamilyToSelectExprs, columnFamilyToStateSourceOptions) =
+          getTimerStateConfigsForCountState(TimeMode.EventTime())
+
+        performRoundTripTest(
+          sourceDir.getAbsolutePath,
+          targetDir.getAbsolutePath,
+          columnFamilyToSchemaMap,
+          columnFamilyToSelectExprs = columnFamilyToSelectExprs,
+          columnFamilyToStateSourceOptions = columnFamilyToStateSourceOptions
+        )
+      }
+    }
+  }
+
+  /**
+   * Helper method to test round-trip for transformWithState with processing time timers.
+   */
+  private def testProcessingTimeTimersRoundTrip(): Unit = {
+    withTempDir { sourceDir =>
+      withTempDir { targetDir =>
+        val clock = new StreamManualClock
+        val inputData = MemoryStream[String]
+        val result = inputData.toDS()
+          .groupByKey(x => x)
+          .transformWithState(new RunningCountStatefulProcessorWithProcTimeTimer(),
+            TimeMode.ProcessingTime(),
+            OutputMode.Update())
+
+        // Step 1: Create source checkpoint
+        testStream(result, OutputMode.Update())(
+          StartStream(checkpointLocation = sourceDir.getAbsolutePath,
+            trigger = Trigger.ProcessingTime("1 second"),
+            triggerClock = clock),
+          AddData(inputData, "a"),
+          AdvanceManualClock(1 * 1000),
+          CheckNewAnswer(("a", "1")),
+          StopStream
+        )
+
+        // Step 2: Create target checkpoint with dummy data
+        val clock2 = new StreamManualClock
+        testStream(result, OutputMode.Update())(
+          StartStream(checkpointLocation = targetDir.getAbsolutePath,
+            trigger = Trigger.ProcessingTime("1 second"),
+            triggerClock = clock2),
+          AddData(inputData, "x"),
+          AdvanceManualClock(1 * 1000),
+          CheckNewAnswer(("a", "1"), ("x", "1")),
+          StopStream
+        )
+
+        // Step 3: Build timer column family configs and perform round-trip test
+        val (columnFamilyToSchemaMap, columnFamilyToSelectExprs, columnFamilyToStateSourceOptions) =
+          getTimerStateConfigsForCountState(TimeMode.ProcessingTime())
+
+        performRoundTripTest(
+          sourceDir.getAbsolutePath,
+          targetDir.getAbsolutePath,
+          columnFamilyToSchemaMap,
+          columnFamilyToSelectExprs = columnFamilyToSelectExprs,
+          columnFamilyToStateSourceOptions = columnFamilyToStateSourceOptions
+        )
+      }
+    }
+  }
+
+  /**
+   * Helper method to test round-trip for transformWithState with list state and TTL.
+   */
+  private def testListStateTTLRoundTrip(): Unit = {
+    withTempDir { sourceDir =>
+      withTempDir { targetDir =>
+        val clock = new StreamManualClock
+        val inputData = MemoryStream[String]
+        val result = inputData.toDS()
+          .groupByKey(x => x)
+          .transformWithState(new ListStateTTLProcessor(),
+            TimeMode.ProcessingTime(),
+            OutputMode.Update())
+
+        // Step 1: Create source checkpoint
+        testStream(result, OutputMode.Update())(
+          StartStream(checkpointLocation = sourceDir.getAbsolutePath,
+            trigger = Trigger.ProcessingTime("1 second"),
+            triggerClock = clock),
+          AddData(inputData, "a", "b", "a"),
+          AdvanceManualClock(1 * 1000),
+          CheckNewAnswer(("a", "2"), ("b", "1")),
+          StopStream
+        )
+
+        // Step 2: Create target checkpoint with dummy data
+        val clock2 = new StreamManualClock
+        testStream(result, OutputMode.Update())(
+          StartStream(checkpointLocation = targetDir.getAbsolutePath,
+            trigger = Trigger.ProcessingTime("1 second"),
+            triggerClock = clock2),
+          AddData(inputData, "x"),
+          AdvanceManualClock(1 * 1000),
+          CheckNewAnswer(("a", "2"), ("b", "1"), ("x", "1")),
+          StopStream
+        )
+
+        // Step 3: Define schemas for list state with TTL column families
+        val groupByKeySchema = StructType(Array(

Review Comment:
   ditto, lets try to see as much things we can move to a common test util. So we can reuse them easily for new test suites



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -676,5 +1151,25 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
         testStreamStreamJoinRoundTrip(version)
       }
     }
+
+    testWithChangelogConfig("SPARK-54411: stream-stream join state ver 3") {
+      testStreamStreamJoinV3RoundTrip()
+    }
+
+    testWithChangelogConfig("SPARK-54411: transformWithState with multiple column families") {

Review Comment:
   for the TWS test cases, lets run them with the different encoding formats i.e. UnsafeRow, Avro. See `AlsoTestWithEncodingTypes`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF [spark]

Posted by "zifeif2 (via GitHub)" <gi...@apache.org>.
zifeif2 commented on code in PR #53386:
URL: https://github.com/apache/spark/pull/53386#discussion_r2633399994


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -123,6 +159,12 @@ class StatePartitionAllColumnFamiliesWriter(
     val valueRow = new UnsafeRow(columnFamilyToValueSchemaLenMap(colFamilyName))
     valueRow.pointTo(valueBytes, valueBytes.length)
 
-    stateStore.put(keyRow, valueRow, colFamilyName)
+    if (columnFamilyToSchemaMap(colFamilyName).useMultipleValuesPerKey) {
+      // if a column family useMultipleValuesPerKey (e.g. ListType), we will
+      // write with 1 put followed by merge
+      stateStore.merge(keyRow, valueRow, colFamilyName)

Review Comment:
   Ok! Will address it in the next version 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF [spark]

Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #53386:
URL: https://github.com/apache/spark/pull/53386#discussion_r2632588138


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -123,6 +159,12 @@ class StatePartitionAllColumnFamiliesWriter(
     val valueRow = new UnsafeRow(columnFamilyToValueSchemaLenMap(colFamilyName))
     valueRow.pointTo(valueBytes, valueBytes.length)
 
-    stateStore.put(keyRow, valueRow, colFamilyName)
+    if (columnFamilyToSchemaMap(colFamilyName).useMultipleValuesPerKey) {
+      // if a column family useMultipleValuesPerKey (e.g. ListType), we will
+      // write with 1 put followed by merge
+      stateStore.merge(keyRow, valueRow, colFamilyName)

Review Comment:
   @zifeif2 merge alone won't work in all cases. The expectation is Put first before merges



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF [spark]

Posted by "zifeif2 (via GitHub)" <gi...@apache.org>.
zifeif2 commented on code in PR #53386:
URL: https://github.com/apache/spark/pull/53386#discussion_r2621206131


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -146,18 +148,47 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
     assert(!checkpointFileExists(new File(targetDir, storeNamePath), versionToCheck, ".changelog"))
     assert(checkpointFileExists(new File(targetDir, storeNamePath), versionToCheck, ".zip"))
 
-    // Step 4: Read from target using normal reader
-    val targetReader = spark.read
-      .format("statestore")
-      .option(StateSourceOptions.PATH, targetDir)
-    val targetNormalData = (storeName match {
-      case Some(name) => targetReader.option(StateSourceOptions.STORE_NAME, name)
-      case None => targetReader
-    }).load()
-      .selectExpr("key", "value", "partition_id")
-      .collect()
+    // Step 3: Validate by reading from both source and target using normal reader"
+    // Default selectExprs for most column families
+    val defaultSelectExprs = Seq("key", "value", "partition_id")
+
+    def shouldCheckColumnFamilyName: String => Boolean = name => {
+      (!name.startsWith("$")
+        || (columnFamilyToStateSourceOptions.contains(name) &&
+        columnFamilyToStateSourceOptions(name).contains(StateSourceOptions.READ_REGISTERED_TIMERS)))
+    }
+    // Validate each column family separately (skip internal column families starting with $)
+    columnFamiliesToValidate
+      // TODO: How to validate that internal columns are written correctly?

Review Comment:
   TODO: will rebase this on latest multi-cf reader and directly query internal column with spark.read...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF [spark]

Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #53386:
URL: https://github.com/apache/spark/pull/53386#discussion_r2636445004


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceTestBase.scala:
##########
@@ -457,3 +458,340 @@ case class SessionUpdate(
     durationMs: Long,
     numEvents: Int,
     expired: Boolean)
+
+case class ColumnFamilyMetadata(
+    keySchema: StructType,
+    valueSchema: StructType,
+    encoderSpec: KeyStateEncoderSpec,
+    useMultipleValuePerKey: Boolean = false)
+
+// Utility for runCompositeKeyStreamingAggregationQuery
+// todo: Move runCompositeKeyStreamingAggregationQuery to this class
+object CompositeKeyAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false),
+      StructField("fruit", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("fruit", StringType, nullable = true),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+// Utility for run DropDuplicatesQueries
+// todo: Move run DropDuplicatesQueries to this class
+object DropDuplicatesTestUtils {
+  def getDropDuplicatesSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", IntegerType, nullable = false),
+      StructField("eventTime", TimestampType)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithColumnSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithColumnSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithColumnSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("col1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType, nullable = true)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithinWatermarkSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("_1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("expiresAtMicros", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for simple streaming aggregation.
+ * Used by StatePartitionAllColumnFamiliesWriterSuite and StatePartitionAllColumnFamiliesReaderSuite
+ * to eliminate code duplication.
+ */
+object SimpleAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version:
+   * @return ColumnFamilyMetadata including schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for flatMapGroupsWithState.
+ */
+object FlatMapGroupsWithStateTestUtils {
+
+  /**
+   * @param stateVersion The state format version:

Review Comment:
   ditto



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceTestBase.scala:
##########
@@ -457,3 +458,340 @@ case class SessionUpdate(
     durationMs: Long,
     numEvents: Int,
     expired: Boolean)
+
+case class ColumnFamilyMetadata(
+    keySchema: StructType,
+    valueSchema: StructType,
+    encoderSpec: KeyStateEncoderSpec,
+    useMultipleValuePerKey: Boolean = false)
+
+// Utility for runCompositeKeyStreamingAggregationQuery
+// todo: Move runCompositeKeyStreamingAggregationQuery to this class
+object CompositeKeyAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false),
+      StructField("fruit", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("fruit", StringType, nullable = true),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+// Utility for run DropDuplicatesQueries
+// todo: Move run DropDuplicatesQueries to this class
+object DropDuplicatesTestUtils {
+  def getDropDuplicatesSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", IntegerType, nullable = false),
+      StructField("eventTime", TimestampType)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithColumnSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithColumnSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithColumnSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("col1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType, nullable = true)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithinWatermarkSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("_1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("expiresAtMicros", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for simple streaming aggregation.
+ * Used by StatePartitionAllColumnFamiliesWriterSuite and StatePartitionAllColumnFamiliesReaderSuite
+ * to eliminate code duplication.
+ */
+object SimpleAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version:
+   * @return ColumnFamilyMetadata including schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for flatMapGroupsWithState.
+ */
+object FlatMapGroupsWithStateTestUtils {
+
+  /**
+   * @param stateVersion The state format version:
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1: Flat structure
+      StructType(Array(
+        StructField("numEvents", IntegerType, nullable = false),
+        StructField("startTimestampMs", LongType, nullable = false),
+        StructField("endTimestampMs", LongType, nullable = false),
+        StructField("timeoutTimestamp", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2: Nested structure with groupState wrapper
+      StructType(Array(
+        StructField("groupState", StructType(Array(
+          StructField("numEvents", IntegerType, nullable = false),
+          StructField("startTimestampMs", LongType, nullable = false),
+          StructField("endTimestampMs", LongType, nullable = false)
+        )), nullable = false),
+        StructField("timeoutTimestamp", LongType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for runSessionWindowAggregationQuery
+ */
+object SessionWindowTestUtils {
+
+  /**
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getSchemas(): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("sessionId", StringType, nullable = false),
+      StructField("sessionStartTime", TimestampType, nullable = false)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("session_window", StructType(Array(
+        StructField("start", TimestampType),
+        StructField("end", TimestampType)
+      )), nullable = false),
+      StructField("sessionId", StringType, nullable = false),
+      StructField("count", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions and constants for runStreamStreamJoinQuery
+ */
+object StreamStreamJoinTestUtils {
+  // Column family names for keyToNumValues stores
+  val KEY_TO_NUM_VALUES_LEFT = "left-keyToNumValues"

Review Comment:
   No need for this, you can call `allStateStoreNames` in `SymmetricHashJoinStateManager` and it will return the list of names



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -47,19 +54,41 @@ class StatePartitionAllColumnFamiliesWriter(
     operatorId: Int,
     storeName: String,
     currentBatchId: Long,
-    columnFamilyToSchemaMap: HashMap[String, StateStoreColFamilySchema]) {
+    colFamilyToWriterInfoMap: Map[String, StatePartitionWriterColumnFamilyInfo],
+    operatorName: String,
+    schemaProviderOpt: Option[StateSchemaProvider],
+    sqlConf: Map[String, String]) {
+
+  private def isJoinV3Operator(
+      operatorName: String, sqlConf: Map[String, String]): Boolean = {

Review Comment:
   ditto



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -47,19 +54,41 @@ class StatePartitionAllColumnFamiliesWriter(
     operatorId: Int,
     storeName: String,
     currentBatchId: Long,
-    columnFamilyToSchemaMap: HashMap[String, StateStoreColFamilySchema]) {
+    colFamilyToWriterInfoMap: Map[String, StatePartitionWriterColumnFamilyInfo],
+    operatorName: String,
+    schemaProviderOpt: Option[StateSchemaProvider],
+    sqlConf: Map[String, String]) {
+
+  private def isJoinV3Operator(
+      operatorName: String, sqlConf: Map[String, String]): Boolean = {
+    operatorName == StatefulOperatorsUtils.SYMMETRIC_HASH_JOIN_EXEC_OP_NAME &&
+      sqlConf(SQLConf.STREAMING_JOIN_STATE_FORMAT_VERSION.key) == "3"
+  }
+
   private val defaultSchema = {
-    columnFamilyToSchemaMap.getOrElse(
-      StateStore.DEFAULT_COL_FAMILY_NAME,
-      throw new IllegalArgumentException(
-        s"Column family ${StateStore.DEFAULT_COL_FAMILY_NAME} not found in schema map")
-    )
+    colFamilyToWriterInfoMap.get(StateStore.DEFAULT_COL_FAMILY_NAME) match {
+      case Some(info) => info.schema
+      case None =>
+        assert(isJoinV3Operator(operatorName, sqlConf),

Review Comment:
   nit: add one line comment, that Join v3 doesn't write default CF schema



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -47,19 +54,41 @@ class StatePartitionAllColumnFamiliesWriter(
     operatorId: Int,
     storeName: String,
     currentBatchId: Long,
-    columnFamilyToSchemaMap: HashMap[String, StateStoreColFamilySchema]) {
+    colFamilyToWriterInfoMap: Map[String, StatePartitionWriterColumnFamilyInfo],
+    operatorName: String,
+    schemaProviderOpt: Option[StateSchemaProvider],
+    sqlConf: Map[String, String]) {

Review Comment:
   nit: take in `SQLConf`



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceTestBase.scala:
##########
@@ -457,3 +458,340 @@ case class SessionUpdate(
     durationMs: Long,
     numEvents: Int,
     expired: Boolean)
+
+case class ColumnFamilyMetadata(
+    keySchema: StructType,
+    valueSchema: StructType,
+    encoderSpec: KeyStateEncoderSpec,
+    useMultipleValuePerKey: Boolean = false)
+
+// Utility for runCompositeKeyStreamingAggregationQuery
+// todo: Move runCompositeKeyStreamingAggregationQuery to this class
+object CompositeKeyAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false),
+      StructField("fruit", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("fruit", StringType, nullable = true),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+// Utility for run DropDuplicatesQueries
+// todo: Move run DropDuplicatesQueries to this class
+object DropDuplicatesTestUtils {
+  def getDropDuplicatesSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", IntegerType, nullable = false),
+      StructField("eventTime", TimestampType)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithColumnSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithColumnSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithColumnSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("col1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType, nullable = true)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithinWatermarkSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("_1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("expiresAtMicros", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for simple streaming aggregation.
+ * Used by StatePartitionAllColumnFamiliesWriterSuite and StatePartitionAllColumnFamiliesReaderSuite
+ * to eliminate code duplication.
+ */
+object SimpleAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version:
+   * @return ColumnFamilyMetadata including schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for flatMapGroupsWithState.
+ */
+object FlatMapGroupsWithStateTestUtils {
+
+  /**
+   * @param stateVersion The state format version:
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1: Flat structure
+      StructType(Array(
+        StructField("numEvents", IntegerType, nullable = false),
+        StructField("startTimestampMs", LongType, nullable = false),
+        StructField("endTimestampMs", LongType, nullable = false),
+        StructField("timeoutTimestamp", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2: Nested structure with groupState wrapper
+      StructType(Array(
+        StructField("groupState", StructType(Array(
+          StructField("numEvents", IntegerType, nullable = false),
+          StructField("startTimestampMs", LongType, nullable = false),
+          StructField("endTimestampMs", LongType, nullable = false)
+        )), nullable = false),
+        StructField("timeoutTimestamp", LongType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for runSessionWindowAggregationQuery
+ */
+object SessionWindowTestUtils {
+
+  /**
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getSchemas(): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("sessionId", StringType, nullable = false),
+      StructField("sessionStartTime", TimestampType, nullable = false)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("session_window", StructType(Array(
+        StructField("start", TimestampType),
+        StructField("end", TimestampType)
+      )), nullable = false),
+      StructField("sessionId", StringType, nullable = false),
+      StructField("count", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions and constants for runStreamStreamJoinQuery
+ */
+object StreamStreamJoinTestUtils {
+  // Column family names for keyToNumValues stores
+  val KEY_TO_NUM_VALUES_LEFT = "left-keyToNumValues"
+  val KEY_TO_NUM_VALUES_RIGHT = "right-keyToNumValues"
+  val KEY_TO_NUM_VALUES_ALL: Seq[String] = Seq(
+    KEY_TO_NUM_VALUES_LEFT,
+    KEY_TO_NUM_VALUES_RIGHT
+  )
+
+  // Column family names for keyWithIndexToValue stores
+  val KEY_WITH_INDEX_LEFT = "left-keyWithIndexToValue"
+  val KEY_WITH_INDEX_RIGHT = "right-keyWithIndexToValue"
+  val KEY_WITH_INDEX_ALL: Seq[String] = Seq(
+    KEY_WITH_INDEX_LEFT,
+    KEY_WITH_INDEX_RIGHT
+  )
+
+  /**
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getKeyToNumValuesSchemas(): (StructType, StructType) = {
+    val metadata = getKeyToNumValuesSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec
+   */
+  def getKeyToNumValuesSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("key", IntegerType)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("value", LongType)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  /**
+   * @param stateVersion The join state format version:
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getKeyWithIndexToValueSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getKeyWithIndexToValueSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec
+   */
+  def getKeyWithIndexToValueSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("key", IntegerType, nullable = false),
+      StructField("index", LongType)
+    ))
+
+    val valueSchema = if (stateVersion == 2 || stateVersion == 3) {
+      StructType(Array(
+        StructField("value", IntegerType, nullable = false),
+        StructField("time", TimestampType, nullable = false),
+        StructField("matched", BooleanType)
+      ))
+    } else {
+      StructType(Array(
+        StructField("value", IntegerType, nullable = false),
+        StructField("time", TimestampType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  /**
+   * Returns all schemas for stream-stream join V3 (multi-column family) in legacy 2-tuple format.
+   * V3 uses a single state store with multiple column families instead of separate stores.
+   *
+   * @return Map of column family name to (keySchema, valueSchema)
+   */
+  def getJoinV3ColumnSchemaMap(): Map[String, (StructType, StructType)] = {
+    getJoinV3ColumnSchemaMapWithMetadata().view.mapValues { metadata =>
+      (metadata.keySchema, metadata.valueSchema)
+    }.toMap
+  }
+
+  /**
+   * @return Map of column family name to ColumnFamilyMetadata
+   */
+  def getJoinV3ColumnSchemaMapWithMetadata(): Map[String, ColumnFamilyMetadata] = {
+    val (keyToNumKeySchema, keyToNumValueSchema) = getKeyToNumValuesSchemas()
+    val (keyWithIndexKeySchema, keyWithIndexValueSchema) = getKeyWithIndexToValueSchemas(3)
+
+    val keyToNumEncoderSpec = NoPrefixKeyStateEncoderSpec(keyToNumKeySchema)
+    val keyWithIndexEncoderSpec = NoPrefixKeyStateEncoderSpec(keyWithIndexKeySchema)
+
+    Map(
+      KEY_TO_NUM_VALUES_LEFT -> ColumnFamilyMetadata(

Review Comment:
   can use `getStateStoreName(Left, KeyToNumValuesType)` in join manager. Same for others



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/utils/TimerTestUtils.scala:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.v2.state.utils
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.Encoders
+import org.apache.spark.sql.execution.streaming.operators.stateful.transformwithstate.timers.TimerStateUtils
+import org.apache.spark.sql.execution.streaming.state.{KeyStateEncoderSpec, NoPrefixKeyStateEncoderSpec, PrefixKeyScanStateEncoderSpec, RangeKeyScanStateEncoderSpec, StateStore}
+import org.apache.spark.sql.streaming.{OutputMode, StatefulProcessor, TimeMode, TimerValues, TTLConfig, ValueState}
+import org.apache.spark.sql.types.{BinaryType, LongType, NullType, StringType, StructField, StructType}
+
+class EventTimeTimerProcessor
+  extends StatefulProcessor[String, (String, Timestamp), (String, String)] {
+  @transient var _valueState: ValueState[Long] = _
+
+  override def init(outputMode: OutputMode, timeMode: TimeMode): Unit = {
+    _valueState = getHandle.getValueState("countState", Encoders.scalaLong, TTLConfig.NONE)
+  }
+
+  override def handleInputRows(
+      key: String,
+      rows: Iterator[(String, Timestamp)],
+      timerValues: TimerValues): Iterator[(String, String)] = {
+    var maxTimestamp = 0L
+    var rowCount = 0
+    rows.foreach { case (_, timestamp) =>
+      maxTimestamp = Math.max(maxTimestamp, timestamp.getTime)
+      rowCount += 1
+    }
+
+    val count = Option(_valueState.get()).getOrElse(0L) + rowCount
+    _valueState.update(count)
+
+    // Register an event time timer
+    if (maxTimestamp > 0) {
+      getHandle.registerTimer(maxTimestamp + 5000)
+    }
+
+    Iterator((key, count.toString))
+  }
+}
+
+/**
+ * Test utility providing schema definitions and constants for EventTimeTimerProcessor
+ * and RunningCountStatefulProcessorWithProcTimeTimer in TransformWithStateSuite
+ */
+object TimerTestUtils {
+  case class ColumnFamilyMetadata(
+      keySchema: StructType,
+      valueSchema: StructType,
+      encoderSpec: KeyStateEncoderSpec,
+      useMultipleValuePerKey: Boolean = false)
+
+  /**
+   * Returns the grouping key schema and state value schema for a simple count state.
+   * This is commonly used with timer tests where state tracks counts.
+   *
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getCountStateSchemas(): (StructType, StructType) = {
+    val groupByKeySchema = StructType(Array(
+      StructField("key", StringType, nullable = true)
+    ))
+    val stateValueSchema = StructType(Array(
+      StructField("value", LongType, nullable = true)
+    ))
+    (groupByKeySchema, stateValueSchema)
+  }
+
+  /**
+   * Returns schemas for timer-related column families
+   *
+   * @param groupingKeySchema The schema for the grouping key
+   * @return A tuple of (keyToTimestampKeySchema, timestampToKeyKeySchema)
+   */
+  def getTimerKeySchemas(groupingKeySchema: StructType): (StructType, StructType) = {
+    val keyToTimestampKeySchema = StructType(Array(
+      StructField("key", groupingKeySchema),
+      StructField("expiryTimestampMs", LongType, nullable = false)
+    ))
+    val timestampToKeyKeySchema = StructType(Array(
+      StructField("expiryTimestampMs", LongType, nullable = false),
+      StructField("key", groupingKeySchema)
+    ))
+
+    (keyToTimestampKeySchema, timestampToKeyKeySchema)
+  }
+
+  /**
+   * Returns complete metadata for timer-related column families including encoder specs.
+   * Used for tests with timers and a count state.
+   * @param timeMode The time mode (EventTime or ProcessingTime)
+   * @return A map from column family name to ColumnFamilyMetadata
+   */
+  def getTimerConfigsForCountState(timeMode: TimeMode): Map[String, ColumnFamilyMetadata] = {
+    val (groupByKeySchema, stateValueSchema) = getCountStateSchemas()
+    val (keyToTimestampKeySchema, timestampToKeyKeySchema) = getTimerKeySchemas(groupByKeySchema)
+    val defaultValueSchema = StructType(Array(StructField("value", BinaryType, nullable = true)))
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(groupByKeySchema)
+    val keyToTimestampEncoderSpec = PrefixKeyScanStateEncoderSpec(keyToTimestampKeySchema, 1)
+    val timestampToKeyEncoderSpec = RangeKeyScanStateEncoderSpec(timestampToKeyKeySchema, Seq(0))
+
+    val (keyToTimestampCF, timestampToKeyCF) =
+      TimerStateUtils.getTimerStateVarNames(timeMode.toString)
+
+    val dummyValueSchema = StructType(Array(StructField("__dummy__", NullType)))
+
+    Map(
+      StateStore.DEFAULT_COL_FAMILY_NAME -> ColumnFamilyMetadata(
+        groupByKeySchema, defaultValueSchema, encoderSpec),
+      "countState" -> ColumnFamilyMetadata(
+        groupByKeySchema, stateValueSchema, encoderSpec),
+      keyToTimestampCF -> ColumnFamilyMetadata(
+        keyToTimestampKeySchema, dummyValueSchema, keyToTimestampEncoderSpec),
+      timestampToKeyCF -> ColumnFamilyMetadata(
+        timestampToKeyKeySchema, dummyValueSchema, timestampToKeyEncoderSpec)
+    )
+  }
+
+  /**
+   * Returns select expressions for timer column families with standardized column order.
+   *
+   * @param columnFamilyName The timer column family name
+   * @return Select expressions as a Seq of strings with order (partition_id, key, value)
+   */
+  def getTimerSelectExpressions(columnFamilyName: String): Seq[String] = {
+    if (columnFamilyName.endsWith("_keyToTimestamp")) {
+      Seq("partition_id",
+          "STRUCT(key AS groupingKey, expiration_timestamp_ms AS key)",
+          "NULL AS value")
+    } else if (columnFamilyName.endsWith("_timestampToKey")) {

Review Comment:
   ditto



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceTestBase.scala:
##########
@@ -457,3 +458,340 @@ case class SessionUpdate(
     durationMs: Long,
     numEvents: Int,
     expired: Boolean)
+
+case class ColumnFamilyMetadata(
+    keySchema: StructType,
+    valueSchema: StructType,
+    encoderSpec: KeyStateEncoderSpec,
+    useMultipleValuePerKey: Boolean = false)
+
+// Utility for runCompositeKeyStreamingAggregationQuery
+// todo: Move runCompositeKeyStreamingAggregationQuery to this class
+object CompositeKeyAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false),
+      StructField("fruit", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("fruit", StringType, nullable = true),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+// Utility for run DropDuplicatesQueries
+// todo: Move run DropDuplicatesQueries to this class
+object DropDuplicatesTestUtils {
+  def getDropDuplicatesSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", IntegerType, nullable = false),
+      StructField("eventTime", TimestampType)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithColumnSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithColumnSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithColumnSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("col1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType, nullable = true)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithinWatermarkSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("_1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("expiresAtMicros", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for simple streaming aggregation.
+ * Used by StatePartitionAllColumnFamiliesWriterSuite and StatePartitionAllColumnFamiliesReaderSuite
+ * to eliminate code duplication.
+ */
+object SimpleAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version:

Review Comment:
   nit: this description gives no valuable info. Can remove it



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceTestBase.scala:
##########
@@ -457,3 +458,340 @@ case class SessionUpdate(
     durationMs: Long,
     numEvents: Int,
     expired: Boolean)
+
+case class ColumnFamilyMetadata(
+    keySchema: StructType,
+    valueSchema: StructType,
+    encoderSpec: KeyStateEncoderSpec,
+    useMultipleValuePerKey: Boolean = false)
+
+// Utility for runCompositeKeyStreamingAggregationQuery
+// todo: Move runCompositeKeyStreamingAggregationQuery to this class
+object CompositeKeyAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false),
+      StructField("fruit", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("fruit", StringType, nullable = true),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+// Utility for run DropDuplicatesQueries
+// todo: Move run DropDuplicatesQueries to this class
+object DropDuplicatesTestUtils {
+  def getDropDuplicatesSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", IntegerType, nullable = false),
+      StructField("eventTime", TimestampType)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithColumnSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithColumnSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithColumnSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("col1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType, nullable = true)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithinWatermarkSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("_1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("expiresAtMicros", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for simple streaming aggregation.
+ * Used by StatePartitionAllColumnFamiliesWriterSuite and StatePartitionAllColumnFamiliesReaderSuite
+ * to eliminate code duplication.
+ */
+object SimpleAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version:
+   * @return ColumnFamilyMetadata including schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for flatMapGroupsWithState.
+ */
+object FlatMapGroupsWithStateTestUtils {
+
+  /**
+   * @param stateVersion The state format version:
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1: Flat structure
+      StructType(Array(
+        StructField("numEvents", IntegerType, nullable = false),
+        StructField("startTimestampMs", LongType, nullable = false),
+        StructField("endTimestampMs", LongType, nullable = false),
+        StructField("timeoutTimestamp", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2: Nested structure with groupState wrapper
+      StructType(Array(
+        StructField("groupState", StructType(Array(
+          StructField("numEvents", IntegerType, nullable = false),
+          StructField("startTimestampMs", LongType, nullable = false),
+          StructField("endTimestampMs", LongType, nullable = false)
+        )), nullable = false),
+        StructField("timeoutTimestamp", LongType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for runSessionWindowAggregationQuery
+ */
+object SessionWindowTestUtils {
+
+  /**
+   * @return A tuple of (keySchema, valueSchema)

Review Comment:
   ditto



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceTestBase.scala:
##########
@@ -457,3 +458,340 @@ case class SessionUpdate(
     durationMs: Long,
     numEvents: Int,
     expired: Boolean)
+
+case class ColumnFamilyMetadata(
+    keySchema: StructType,
+    valueSchema: StructType,
+    encoderSpec: KeyStateEncoderSpec,
+    useMultipleValuePerKey: Boolean = false)
+
+// Utility for runCompositeKeyStreamingAggregationQuery
+// todo: Move runCompositeKeyStreamingAggregationQuery to this class
+object CompositeKeyAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false),
+      StructField("fruit", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("fruit", StringType, nullable = true),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+// Utility for run DropDuplicatesQueries
+// todo: Move run DropDuplicatesQueries to this class
+object DropDuplicatesTestUtils {
+  def getDropDuplicatesSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", IntegerType, nullable = false),
+      StructField("eventTime", TimestampType)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithColumnSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithColumnSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithColumnSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("col1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType, nullable = true)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithinWatermarkSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("_1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("expiresAtMicros", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for simple streaming aggregation.
+ * Used by StatePartitionAllColumnFamiliesWriterSuite and StatePartitionAllColumnFamiliesReaderSuite
+ * to eliminate code duplication.
+ */
+object SimpleAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version:
+   * @return ColumnFamilyMetadata including schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for flatMapGroupsWithState.
+ */
+object FlatMapGroupsWithStateTestUtils {
+
+  /**
+   * @param stateVersion The state format version:
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1: Flat structure
+      StructType(Array(
+        StructField("numEvents", IntegerType, nullable = false),
+        StructField("startTimestampMs", LongType, nullable = false),
+        StructField("endTimestampMs", LongType, nullable = false),
+        StructField("timeoutTimestamp", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2: Nested structure with groupState wrapper
+      StructType(Array(
+        StructField("groupState", StructType(Array(
+          StructField("numEvents", IntegerType, nullable = false),
+          StructField("startTimestampMs", LongType, nullable = false),
+          StructField("endTimestampMs", LongType, nullable = false)
+        )), nullable = false),
+        StructField("timeoutTimestamp", LongType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for runSessionWindowAggregationQuery
+ */
+object SessionWindowTestUtils {
+
+  /**
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getSchemas(): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec

Review Comment:
   ditto, do for the remaining too



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -123,6 +173,16 @@ class StatePartitionAllColumnFamiliesWriter(
     val valueRow = new UnsafeRow(columnFamilyToValueSchemaLenMap(colFamilyName))
     valueRow.pointTo(valueBytes, valueBytes.length)
 
-    stateStore.put(keyRow, valueRow, colFamilyName)
+    if (colFamilyToWriterInfoMap(colFamilyName).useMultipleValuesPerKey) {
+      // if a column family useMultipleValuesPerKey (e.g. ListType), we will
+      // write with 1 put followed by merge
+      if (stateStore.keyExists(keyRow, colFamilyName)) {

Review Comment:
   These 2 ifs can be combined into one i.e.
   ```
   // if a column family useMultipleValuesPerKey (e.g. ListType), we will
   // write with 1 put followed by merge
   if (colFamilyToWriterInfoMap(colFamilyName).useMultipleValuesPerKey && 
     stateStore.keyExists(keyRow, colFamilyName)) {
     stateStore.merge(keyRow, valueRow, colFamilyName)
   } else {
     stateStore.put(keyRow, valueRow, colFamilyName)
   }
   ```



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceTestBase.scala:
##########
@@ -457,3 +458,340 @@ case class SessionUpdate(
     durationMs: Long,
     numEvents: Int,
     expired: Boolean)
+
+case class ColumnFamilyMetadata(
+    keySchema: StructType,
+    valueSchema: StructType,
+    encoderSpec: KeyStateEncoderSpec,
+    useMultipleValuePerKey: Boolean = false)
+
+// Utility for runCompositeKeyStreamingAggregationQuery
+// todo: Move runCompositeKeyStreamingAggregationQuery to this class
+object CompositeKeyAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false),
+      StructField("fruit", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("fruit", StringType, nullable = true),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+// Utility for run DropDuplicatesQueries
+// todo: Move run DropDuplicatesQueries to this class
+object DropDuplicatesTestUtils {
+  def getDropDuplicatesSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", IntegerType, nullable = false),
+      StructField("eventTime", TimestampType)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithColumnSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithColumnSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithColumnSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("col1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType, nullable = true)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithinWatermarkSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("_1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("expiresAtMicros", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for simple streaming aggregation.
+ * Used by StatePartitionAllColumnFamiliesWriterSuite and StatePartitionAllColumnFamiliesReaderSuite
+ * to eliminate code duplication.
+ */
+object SimpleAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version:
+   * @return ColumnFamilyMetadata including schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for flatMapGroupsWithState.
+ */
+object FlatMapGroupsWithStateTestUtils {
+
+  /**
+   * @param stateVersion The state format version:
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version

Review Comment:
   ditto



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/utils/TimerTestUtils.scala:
##########
@@ -0,0 +1,170 @@
+/*

Review Comment:
   Lets combine these 3 utils into one file. I mean the MultiStateVarTestUtils file + TimerTestUtils + TTLProcessorUtils. No need for separate files, they are all for TWS testing and doing similar things. Makes them easier to find.



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/utils/TimerTestUtils.scala:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.v2.state.utils
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.Encoders
+import org.apache.spark.sql.execution.streaming.operators.stateful.transformwithstate.timers.TimerStateUtils
+import org.apache.spark.sql.execution.streaming.state.{KeyStateEncoderSpec, NoPrefixKeyStateEncoderSpec, PrefixKeyScanStateEncoderSpec, RangeKeyScanStateEncoderSpec, StateStore}
+import org.apache.spark.sql.streaming.{OutputMode, StatefulProcessor, TimeMode, TimerValues, TTLConfig, ValueState}
+import org.apache.spark.sql.types.{BinaryType, LongType, NullType, StringType, StructField, StructType}
+
+class EventTimeTimerProcessor
+  extends StatefulProcessor[String, (String, Timestamp), (String, String)] {
+  @transient var _valueState: ValueState[Long] = _
+
+  override def init(outputMode: OutputMode, timeMode: TimeMode): Unit = {
+    _valueState = getHandle.getValueState("countState", Encoders.scalaLong, TTLConfig.NONE)
+  }
+
+  override def handleInputRows(
+      key: String,
+      rows: Iterator[(String, Timestamp)],
+      timerValues: TimerValues): Iterator[(String, String)] = {
+    var maxTimestamp = 0L
+    var rowCount = 0
+    rows.foreach { case (_, timestamp) =>
+      maxTimestamp = Math.max(maxTimestamp, timestamp.getTime)
+      rowCount += 1
+    }
+
+    val count = Option(_valueState.get()).getOrElse(0L) + rowCount
+    _valueState.update(count)
+
+    // Register an event time timer
+    if (maxTimestamp > 0) {
+      getHandle.registerTimer(maxTimestamp + 5000)
+    }
+
+    Iterator((key, count.toString))
+  }
+}
+
+/**
+ * Test utility providing schema definitions and constants for EventTimeTimerProcessor
+ * and RunningCountStatefulProcessorWithProcTimeTimer in TransformWithStateSuite
+ */
+object TimerTestUtils {
+  case class ColumnFamilyMetadata(

Review Comment:
   this is defined in several places. Just define it in one place



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/utils/TimerTestUtils.scala:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.v2.state.utils
+
+import java.sql.Timestamp
+
+import org.apache.spark.sql.Encoders
+import org.apache.spark.sql.execution.streaming.operators.stateful.transformwithstate.timers.TimerStateUtils
+import org.apache.spark.sql.execution.streaming.state.{KeyStateEncoderSpec, NoPrefixKeyStateEncoderSpec, PrefixKeyScanStateEncoderSpec, RangeKeyScanStateEncoderSpec, StateStore}
+import org.apache.spark.sql.streaming.{OutputMode, StatefulProcessor, TimeMode, TimerValues, TTLConfig, ValueState}
+import org.apache.spark.sql.types.{BinaryType, LongType, NullType, StringType, StructField, StructType}
+
+class EventTimeTimerProcessor
+  extends StatefulProcessor[String, (String, Timestamp), (String, String)] {
+  @transient var _valueState: ValueState[Long] = _
+
+  override def init(outputMode: OutputMode, timeMode: TimeMode): Unit = {
+    _valueState = getHandle.getValueState("countState", Encoders.scalaLong, TTLConfig.NONE)
+  }
+
+  override def handleInputRows(
+      key: String,
+      rows: Iterator[(String, Timestamp)],
+      timerValues: TimerValues): Iterator[(String, String)] = {
+    var maxTimestamp = 0L
+    var rowCount = 0
+    rows.foreach { case (_, timestamp) =>
+      maxTimestamp = Math.max(maxTimestamp, timestamp.getTime)
+      rowCount += 1
+    }
+
+    val count = Option(_valueState.get()).getOrElse(0L) + rowCount
+    _valueState.update(count)
+
+    // Register an event time timer
+    if (maxTimestamp > 0) {
+      getHandle.registerTimer(maxTimestamp + 5000)
+    }
+
+    Iterator((key, count.toString))
+  }
+}
+
+/**
+ * Test utility providing schema definitions and constants for EventTimeTimerProcessor
+ * and RunningCountStatefulProcessorWithProcTimeTimer in TransformWithStateSuite
+ */
+object TimerTestUtils {
+  case class ColumnFamilyMetadata(
+      keySchema: StructType,
+      valueSchema: StructType,
+      encoderSpec: KeyStateEncoderSpec,
+      useMultipleValuePerKey: Boolean = false)
+
+  /**
+   * Returns the grouping key schema and state value schema for a simple count state.
+   * This is commonly used with timer tests where state tracks counts.
+   *
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getCountStateSchemas(): (StructType, StructType) = {
+    val groupByKeySchema = StructType(Array(
+      StructField("key", StringType, nullable = true)
+    ))
+    val stateValueSchema = StructType(Array(
+      StructField("value", LongType, nullable = true)
+    ))
+    (groupByKeySchema, stateValueSchema)
+  }
+
+  /**
+   * Returns schemas for timer-related column families
+   *
+   * @param groupingKeySchema The schema for the grouping key
+   * @return A tuple of (keyToTimestampKeySchema, timestampToKeyKeySchema)
+   */
+  def getTimerKeySchemas(groupingKeySchema: StructType): (StructType, StructType) = {
+    val keyToTimestampKeySchema = StructType(Array(
+      StructField("key", groupingKeySchema),
+      StructField("expiryTimestampMs", LongType, nullable = false)
+    ))
+    val timestampToKeyKeySchema = StructType(Array(
+      StructField("expiryTimestampMs", LongType, nullable = false),
+      StructField("key", groupingKeySchema)
+    ))
+
+    (keyToTimestampKeySchema, timestampToKeyKeySchema)
+  }
+
+  /**
+   * Returns complete metadata for timer-related column families including encoder specs.
+   * Used for tests with timers and a count state.
+   * @param timeMode The time mode (EventTime or ProcessingTime)
+   * @return A map from column family name to ColumnFamilyMetadata
+   */
+  def getTimerConfigsForCountState(timeMode: TimeMode): Map[String, ColumnFamilyMetadata] = {
+    val (groupByKeySchema, stateValueSchema) = getCountStateSchemas()
+    val (keyToTimestampKeySchema, timestampToKeyKeySchema) = getTimerKeySchemas(groupByKeySchema)
+    val defaultValueSchema = StructType(Array(StructField("value", BinaryType, nullable = true)))
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(groupByKeySchema)
+    val keyToTimestampEncoderSpec = PrefixKeyScanStateEncoderSpec(keyToTimestampKeySchema, 1)
+    val timestampToKeyEncoderSpec = RangeKeyScanStateEncoderSpec(timestampToKeyKeySchema, Seq(0))
+
+    val (keyToTimestampCF, timestampToKeyCF) =
+      TimerStateUtils.getTimerStateVarNames(timeMode.toString)
+
+    val dummyValueSchema = StructType(Array(StructField("__dummy__", NullType)))
+
+    Map(
+      StateStore.DEFAULT_COL_FAMILY_NAME -> ColumnFamilyMetadata(
+        groupByKeySchema, defaultValueSchema, encoderSpec),
+      "countState" -> ColumnFamilyMetadata(
+        groupByKeySchema, stateValueSchema, encoderSpec),
+      keyToTimestampCF -> ColumnFamilyMetadata(
+        keyToTimestampKeySchema, dummyValueSchema, keyToTimestampEncoderSpec),
+      timestampToKeyCF -> ColumnFamilyMetadata(
+        timestampToKeyKeySchema, dummyValueSchema, timestampToKeyEncoderSpec)
+    )
+  }
+
+  /**
+   * Returns select expressions for timer column families with standardized column order.
+   *
+   * @param columnFamilyName The timer column family name
+   * @return Select expressions as a Seq of strings with order (partition_id, key, value)
+   */
+  def getTimerSelectExpressions(columnFamilyName: String): Seq[String] = {
+    if (columnFamilyName.endsWith("_keyToTimestamp")) {

Review Comment:
   This is already defined in `TimerStateUtils.KEY_TO_TIMESTAMP_CF`



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceTestBase.scala:
##########
@@ -457,3 +458,340 @@ case class SessionUpdate(
     durationMs: Long,
     numEvents: Int,
     expired: Boolean)
+
+case class ColumnFamilyMetadata(
+    keySchema: StructType,
+    valueSchema: StructType,
+    encoderSpec: KeyStateEncoderSpec,
+    useMultipleValuePerKey: Boolean = false)
+
+// Utility for runCompositeKeyStreamingAggregationQuery
+// todo: Move runCompositeKeyStreamingAggregationQuery to this class
+object CompositeKeyAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false),
+      StructField("fruit", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("fruit", StringType, nullable = true),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+// Utility for run DropDuplicatesQueries
+// todo: Move run DropDuplicatesQueries to this class
+object DropDuplicatesTestUtils {
+  def getDropDuplicatesSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", IntegerType, nullable = false),
+      StructField("eventTime", TimestampType)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithColumnSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithColumnSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithColumnSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("col1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("__dummy__", NullType, nullable = true)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemas(): (StructType, StructType) = {
+    val metadata = getDropDuplicatesWithinWatermarkSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  def getDropDuplicatesWithinWatermarkSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("_1", StringType, nullable = true)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("expiresAtMicros", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for simple streaming aggregation.
+ * Used by StatePartitionAllColumnFamiliesWriterSuite and StatePartitionAllColumnFamiliesReaderSuite
+ * to eliminate code duplication.
+ */
+object SimpleAggregationTestUtils {
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version:
+   * @return ColumnFamilyMetadata including schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("groupKey", IntegerType, nullable = false)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1 includes key columns in the value
+      StructType(Array(
+        StructField("groupKey", IntegerType, nullable = false),
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2 excludes key columns from the value
+      StructType(Array(
+        StructField("count", LongType, nullable = false),
+        StructField("sum", LongType, nullable = false),
+        StructField("max", IntegerType, nullable = false),
+        StructField("min", IntegerType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for flatMapGroupsWithState.
+ */
+object FlatMapGroupsWithStateTestUtils {
+
+  /**
+   * @param stateVersion The state format version:
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getSchemas(stateVersion: Int): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata(stateVersion)
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @param stateVersion The state format version
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(stateVersion: Int): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("value", StringType, nullable = true)
+    ))
+
+    val valueSchema = if (stateVersion == 1) {
+      // State version 1: Flat structure
+      StructType(Array(
+        StructField("numEvents", IntegerType, nullable = false),
+        StructField("startTimestampMs", LongType, nullable = false),
+        StructField("endTimestampMs", LongType, nullable = false),
+        StructField("timeoutTimestamp", IntegerType, nullable = false)
+      ))
+    } else {
+      // State version 2: Nested structure with groupState wrapper
+      StructType(Array(
+        StructField("groupState", StructType(Array(
+          StructField("numEvents", IntegerType, nullable = false),
+          StructField("startTimestampMs", LongType, nullable = false),
+          StructField("endTimestampMs", LongType, nullable = false)
+        )), nullable = false),
+        StructField("timeoutTimestamp", LongType, nullable = false)
+      ))
+    }
+
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions for runSessionWindowAggregationQuery
+ */
+object SessionWindowTestUtils {
+
+  /**
+   * @return A tuple of (keySchema, valueSchema)
+   */
+  def getSchemas(): (StructType, StructType) = {
+    val metadata = getSchemasWithMetadata()
+    (metadata.keySchema, metadata.valueSchema)
+  }
+
+  /**
+   * @return ColumnFamilyMetadata with schema and KeyEncoderSpec
+   */
+  def getSchemasWithMetadata(): ColumnFamilyMetadata = {
+    val keySchema = StructType(Array(
+      StructField("sessionId", StringType, nullable = false),
+      StructField("sessionStartTime", TimestampType, nullable = false)
+    ))
+    val valueSchema = StructType(Array(
+      StructField("session_window", StructType(Array(
+        StructField("start", TimestampType),
+        StructField("end", TimestampType)
+      )), nullable = false),
+      StructField("sessionId", StringType, nullable = false),
+      StructField("count", LongType, nullable = false)
+    ))
+    val encoderSpec = NoPrefixKeyStateEncoderSpec(keySchema)
+    ColumnFamilyMetadata(keySchema, valueSchema, encoderSpec)
+  }
+}
+
+/**
+ * Test utility object providing schema definitions and constants for runStreamStreamJoinQuery
+ */
+object StreamStreamJoinTestUtils {
+  // Column family names for keyToNumValues stores
+  val KEY_TO_NUM_VALUES_LEFT = "left-keyToNumValues"
+  val KEY_TO_NUM_VALUES_RIGHT = "right-keyToNumValues"
+  val KEY_TO_NUM_VALUES_ALL: Seq[String] = Seq(
+    KEY_TO_NUM_VALUES_LEFT,
+    KEY_TO_NUM_VALUES_RIGHT
+  )
+
+  // Column family names for keyWithIndexToValue stores
+  val KEY_WITH_INDEX_LEFT = "left-keyWithIndexToValue"

Review Comment:
   ditto



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF [spark]

Posted by "zifeif2 (via GitHub)" <gi...@apache.org>.
zifeif2 commented on code in PR #53386:
URL: https://github.com/apache/spark/pull/53386#discussion_r2625691797


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -329,15 +389,42 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
           performRoundTripTest(
             sourceDir.getAbsolutePath,
             targetDir.getAbsolutePath,
-            keySchema,
-            valueSchema,
-            keyStateEncoderSpec
+            createSingleColumnFamilySchemaMap(keySchema, valueSchema, keyStateEncoderSpec)
           )
         }
       }
     }
   }
 
+  private val keyToNumValuesColFamilyNames = Seq("left-keyToNumValues", "right-keyToNumValues")

Review Comment:
   They are the same! Okay I'll move them somewhere



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionWriter.scala:
##########
@@ -123,6 +159,12 @@ class StatePartitionAllColumnFamiliesWriter(
     val valueRow = new UnsafeRow(columnFamilyToValueSchemaLenMap(colFamilyName))
     valueRow.pointTo(valueBytes, valueBytes.length)
 
-    stateStore.put(keyRow, valueRow, colFamilyName)
+    if (columnFamilyToSchemaMap(colFamilyName).useMultipleValuesPerKey) {
+      // if a column family useMultipleValuesPerKey (e.g. ListType), we will
+      // write with 1 put followed by merge
+      stateStore.merge(keyRow, valueRow, colFamilyName)

Review Comment:
   Ah I should probably update the comment but I tried just using merge and the test was correct? 



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StatePartitionAllColumnFamiliesWriterSuite.scala:
##########
@@ -146,18 +148,47 @@ class StatePartitionAllColumnFamiliesWriterSuite extends StateDataSourceTestBase
     assert(!checkpointFileExists(new File(targetDir, storeNamePath), versionToCheck, ".changelog"))
     assert(checkpointFileExists(new File(targetDir, storeNamePath), versionToCheck, ".zip"))
 
-    // Step 4: Read from target using normal reader
-    val targetReader = spark.read
-      .format("statestore")
-      .option(StateSourceOptions.PATH, targetDir)
-    val targetNormalData = (storeName match {
-      case Some(name) => targetReader.option(StateSourceOptions.STORE_NAME, name)
-      case None => targetReader
-    }).load()
-      .selectExpr("key", "value", "partition_id")
-      .collect()
+    // Step 3: Validate by reading from both source and target using normal reader"
+    // Default selectExprs for most column families
+    val defaultSelectExprs = Seq("key", "value", "partition_id")
+
+    def shouldCheckColumnFamilyName: String => Boolean = name => {
+      (!name.startsWith("$")
+        || (columnFamilyToStateSourceOptions.contains(name) &&

Review Comment:
   We won't need this anymore because the latest multi-cf reader PR supports reading internal columns in testing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org