You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HeartSaVioR (via GitHub)" <gi...@apache.org> on 2026/03/19 06:10:19 UTC

Re: [PR] [SPARK-55147][SS] Scope timestamp range for time-interval join retrieval in V4 state format [spark]

HeartSaVioR commented on code in PR #54879:
URL: https://github.com/apache/spark/pull/54879#discussion_r2957783008


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:
##########
@@ -67,11 +67,15 @@ trait SymmetricHashJoinStateManager {
    * required to do so.
    *
    * It is caller's responsibility to consume the whole iterator.
+   *
+   * For V4 time-interval joins, timestampRange may be provided to skip/stop-early during

Review Comment:
   Let's keep the documentation of interface to take a role of "interface". We can just talk about contract/requirement and derived classes will have separate method doc to mention the fact.
   
   I'd rather document the parameter `timestampRange` as "hint" for optimization of reducing scope of scan. The derived class can make an optimization with that hint but it's still OK for derived class to ignore it, if the derived class cannot leverage that hint. 
   
   We should still also clarify that given `timestampRange` is a hint and derived class can decide not to leverage it, `timestampRange` is expected to be a subset of `predicate` condition in practice. That means, the parameter `predicate` has to be provided in a way to produce the correct output whether the parameter `timestampRange` is leveraged as a hint or not.
   
   (The reason I said "in practice" is because you leverage this in the test and I admit there is no easy way to test this except breaking the above.)



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:
##########
@@ -626,67 +630,86 @@ class SymmetricHashJoinStateManagerV4(
 
     // NOTE: This assumes we consume the whole iterator to trigger completion.
     def getValues(key: UnsafeRow): Iterator[GetValuesResult] = {
+      getValuesInRange(key, Long.MinValue, Long.MaxValue)
+    }
+
+    /**
+     * Returns entries where minTs <= timestamp <= maxTs, grouped by timestamp.
+     * Filters out entries before minTs and stops iterating past maxTs (timestamps are sorted).
+     */
+    def getValuesInRange(
+        key: UnsafeRow, minTs: Long, maxTs: Long): Iterator[GetValuesResult] = {
       val reusableGetValuesResult = new GetValuesResult()
 
       new NextIterator[GetValuesResult] {
         private val iter = stateStore.prefixScanWithMultiValues(key, colFamilyName)
 
         private var currentTs = -1L
+        private var currentTsInRange = false

Review Comment:
   Honestly the new logic is over complicated. Technically, the entire logic should be the same as before, except handling lower bound and upper bound. Handling lower bound and upper bound can be handled as exceptional case than processing the data in timestamp boundary.
   
   Below is the simplified logic (DISCLAIMER: Claude 4.6 opus) with my guidance of direction for above simplification:
   
   ```
           private var currentTs = -1L
           private var pastUpperBound = false
           private val valueAndMatchPairs = scala.collection.mutable.ArrayBuffer[ValueAndMatchPair]()
   
           private def flushAccumulated(): GetValuesResult = {
             if (valueAndMatchPairs.nonEmpty) {
               val result = reusableGetValuesResult.withNew(
                 currentTs, valueAndMatchPairs.toList)
               currentTs = -1L
               valueAndMatchPairs.clear()
               result
             } else {
               finished = true
               null
             }
           }
   
           @tailrec
           override protected def getNext(): GetValuesResult = {
             if (pastUpperBound || !iter.hasNext) {
               flushAccumulated()
             } else {
               val unsafeRowPair = iter.next()
               val ts = TimestampKeyStateEncoder.extractTimestamp(unsafeRowPair.key)
   
               if (ts > maxTs) {
                 pastUpperBound = true
                 getNext()
               } else if (ts < minTs) {
                 getNext()
               } else if (currentTs == -1L || currentTs == ts) {
                 currentTs = ts
                 valueAndMatchPairs += valueRowConverter.convertValue(unsafeRowPair.value)
                 getNext()
               } else {
                 // Timestamp changed -- flush previous group before starting new one
                 val prevTs = currentTs
                 val prevValues = valueAndMatchPairs.toList
   
                 currentTs = ts
                 valueAndMatchPairs.clear()
                 valueAndMatchPairs += valueRowConverter.convertValue(unsafeRowPair.value)
   
                 reusableGetValuesResult.withNew(prevTs, prevValues)
               }
             }
           }
   ```
   
   Would you take a look and apply the change if you think it's good?



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:
##########
@@ -626,67 +630,86 @@ class SymmetricHashJoinStateManagerV4(
 
     // NOTE: This assumes we consume the whole iterator to trigger completion.
     def getValues(key: UnsafeRow): Iterator[GetValuesResult] = {
+      getValuesInRange(key, Long.MinValue, Long.MaxValue)
+    }
+
+    /**
+     * Returns entries where minTs <= timestamp <= maxTs, grouped by timestamp.
+     * Filters out entries before minTs and stops iterating past maxTs (timestamps are sorted).
+     */
+    def getValuesInRange(
+        key: UnsafeRow, minTs: Long, maxTs: Long): Iterator[GetValuesResult] = {
       val reusableGetValuesResult = new GetValuesResult()
 
       new NextIterator[GetValuesResult] {
         private val iter = stateStore.prefixScanWithMultiValues(key, colFamilyName)
 
         private var currentTs = -1L
+        private var currentTsInRange = false
+        private var pastUpperBound = false
         private val valueAndMatchPairs = scala.collection.mutable.ArrayBuffer[ValueAndMatchPair]()
 
+        private def flushIfInRange(): GetValuesResult = {
+          if (currentTsInRange && valueAndMatchPairs.nonEmpty) {
+            val result = reusableGetValuesResult.withNew(
+              currentTs, valueAndMatchPairs.toList)
+            currentTs = -1L
+            currentTsInRange = false
+            valueAndMatchPairs.clear()
+            result
+          } else {
+            finished = true
+            null
+          }
+        }
+
         @tailrec
         override protected def getNext(): GetValuesResult = {
-          if (iter.hasNext) {
+          if (pastUpperBound) {
+            flushIfInRange()
+          } else if (iter.hasNext) {
             val unsafeRowPair = iter.next()
-
             val ts = TimestampKeyStateEncoder.extractTimestamp(unsafeRowPair.key)

Review Comment:
   I think this can be considered as "beyond the scope of PR" (since it requires some code change on StateStore API), but just for recording:
   
   We might get better outcome if we can specify the start boundary and the end boundary when scanning in general. This would be useful for range scan and prefix scan with timestamp (timestamp as postfix).
   
   Seeking to specific position directly is better than seeking to first position and doing sequential scan to find the specific position. (RocksDB won't find the position via sequentially scan)
   
   Scoping the iterator to closely in upper bound would also help RocksDB to avoid unnecessary scanning (especially tombstones), although it's not probably very different due to the pattern we remove the state (we remove the state in timestamp order, so it's unlikely to have tombstones beyond upper bound if we ever have a valid entry within timestamp boundary.



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManagerSuite.scala:
##########
@@ -1009,4 +1009,48 @@ class SymmetricHashJoinStateManagerEventTimeInValueSuite
       }
     }
   }
+
+  private def getJoinedRowTimestamps(
+      key: Int,
+      range: Option[(Long, Long)])(implicit manager: SymmetricHashJoinStateManager): Seq[Int] = {
+    val dummyRow = new GenericInternalRow(0)
+    manager.getJoinedRows(

Review Comment:
   Let's briefly leave code comment that `predicate` should contain the condition of `timestampRange` in practice, but we don't do it intentionally for testing the functionality.



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:
##########
@@ -67,11 +67,15 @@ trait SymmetricHashJoinStateManager {
    * required to do so.
    *
    * It is caller's responsibility to consume the whole iterator.
+   *
+   * For V4 time-interval joins, timestampRange may be provided to skip/stop-early during

Review Comment:
   Also let's clarify the boundary of inclusive vs exclusive in both sides. This should be described in the interface method doc.



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManagerSuite.scala:
##########
@@ -1009,4 +1009,48 @@ class SymmetricHashJoinStateManagerEventTimeInValueSuite
       }
     }
   }
+
+  private def getJoinedRowTimestamps(
+      key: Int,
+      range: Option[(Long, Long)])(implicit manager: SymmetricHashJoinStateManager): Seq[Int] = {
+    val dummyRow = new GenericInternalRow(0)
+    manager.getJoinedRows(
+      toJoinKeyRow(key),
+      row => new JoinedRow(row, dummyRow),
+      _ => true,
+      timestampRange = range
+    ).map(_.getInt(1)).toSeq.sorted
+  }
+
+  test("StreamingJoinStateManager V4 - getJoinedRows with timestampRange") {
+    withJoinStateManager(
+      inputValueAttributes, joinKeyExpressions, stateFormatVersion = 4) { manager =>
+      implicit val mgr = manager
+
+      Seq(10, 20, 30, 40, 50).foreach(append(40, _))
+
+      assert(getJoinedRowTimestamps(40, Some((20L, 40L))) === Seq(20, 30, 40))
+      assert(getJoinedRowTimestamps(40, Some((20L, 20L))) === Seq(20))
+      assert(getJoinedRowTimestamps(40, Some((25L, 35L))) === Seq(30))
+      assert(getJoinedRowTimestamps(40, Some((0L, 100L))) === Seq(10, 20, 30, 40, 50))
+      assert(getJoinedRowTimestamps(40, Some((10L, 30L))) === Seq(10, 20, 30))
+      assert(getJoinedRowTimestamps(40, Some((50L, 100L))) === Seq(50))
+      assert(getJoinedRowTimestamps(40, Some((60L, 100L))) === Seq.empty)
+      assert(getJoinedRowTimestamps(40, Some((0L, 5L))) === Seq.empty)
+      assert(getJoinedRowTimestamps(40, None) === Seq(10, 20, 30, 40, 50))
+    }
+  }
+
+  test("StreamingJoinStateManager V4 - timestampRange with multiple values per timestamp") {
+    withJoinStateManager(
+      inputValueAttributes, joinKeyExpressions, stateFormatVersion = 4) { manager =>
+      implicit val mgr = manager
+
+      append(40, 20)

Review Comment:
   Why not adding a bit more to show that the method works correctly with multiple values across multiple timestamp buckets?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org