You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HeartSaVioR (via GitHub)" <gi...@apache.org> on 2023/02/07 15:14:23 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request, #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
HeartSaVioR opened a new pull request, #39931:
URL: https://github.com/apache/spark/pull/39931
### What changes were proposed in this pull request?
This PR proposes to introduce watermark propagation among operators via simulation, which enables the workload of "stream-stream time interval join followed by stateful operator".
As of now, Spark considers all stateful operators to have same input watermark and output watermark, which is insufficient to handle stream-stream time interval join. It can delay joined output more than global watermark, based on the join criteria. (e.g. `leftTime BETWEEN rightTime - INTERVAL 30 seconds AND rightTime + INTERVAL 40 seconds`). To address this, the join operator should be able to produce "delayed" watermark to the downstream operator. That said, Spark has to "propagate" watermark among operators, flowing through leaf node(s) to root (going downstream).
This PR introduces a new interface `WatermarkPropagator` which performs simulation of watermark propagation based on the watermark. There are three implementations for this interface:
1. NoOpWatermarkPropagator: Do nothing. This is used for initializing dummy IncrementalExecution.
2. UseSingleWatermarkPropagator: Uses a single global watermark for late events and eviction. This is used for compatibility mode (`spark.sql.streaming.statefulOperator.allowMultiple` to `false`).
3. PropagateWatermarkSimulator: simulates propagation of watermark among operators.
The simulation algorithm used in `PropagateWatermarkSimulator` traverses the physical plan tree via post-order (children first) to calculate (input watermark, output watermark) for all nodes. For each node, below logic is applied:
- Input watermark for specific node is decided by `min(input watermarks from all children)`.
- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
- Output watermark for specific node is decided as following:
- watermark nodes: origin watermark value (global watermark).
- stateless nodes: same as input watermark.
- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
Once the algorithm traverses the physical plan tree, the association between stateful operator and input watermark will be constructed. The association is cached after calculation and being used across microbatches, till Spark determines the association as no longer to be used.
As mentioned like `op.produceWatermark()` in above, this PR also adds a new method `produceWatermark` in StateStoreWriter, which requires each stateful operator to calculate output watermark based on given input watermark. In most cases, this is same as the criteria of state eviction, as most stateful operators produce the output from two different kinds:
1. without buffering (event time > input watermark)
2. with buffering (state)
The state eviction happens when event time exceeds a "certain threshold of timestamp", which denotes a lower bound of event time values for output (output watermark). Since most stateful operators construct the predicate for state eviction based on watermark in batch planning phase, they can produce an output watermark once Spark provides an input watermark.
Please refer to the walkthrough code comment for the test case of `stream-stream time interval left outer join -> aggregation, append mode`.
There are several additional decisions made by this PR which introduces backward incompatibility.
1. Re-definition of watermark will be disallowed.
Technically, each watermark node can track its own value of watermark and PropagateWatermarkSimulator can propagate these values correctly. (multiple origins) While this may help to accelerate processing faster stream (as all watermarks don't need to follow the slowest one till join/union), this involves more complicated questions on UX perspective, as all UX about watermark is based on global watermark. This seems harder to address, hence this PR proposes to retain the global watermark as it is.
Since we want to produce watermark as the single origin value, redefinition of watermark does not make sense. Consider stream-stream time interval join followed by another watermark node. Which is the right value of output watermark for another watermark node? delayed watermark, or global watermark?
2. stateful operator will not allow multiple event time columns being defined in the input DataFrame.
The output of stream-stream join may have two event time columns, which is ambiguous on late record filtering and eviction. Currently the first appeared column has been picked up for late record filtering and eviction, which is NOT correct. After this PR, Spark will throw an exception.
**TODO: Apply this to the code change.**
Turning off the flag `spark.sql.streaming.statefulOperator.allowMultiple` will restore the old behavior from the above.
### Why are the changes needed?
stream-stream time interval join followed by stateful operator is not supported yet.
### Does this PR introduce _any_ user-facing change?
Yes, here is a list of user facing changes (some are backward incompatibility changes, though we have compatibility flag):
- stream-stream time-interval join followed by stateful operator will be allowed.
- Re-definition of watermark will be disallowed.
- stateful operator will not allow multiple event time columns being defined in the input DataFrame.
### How was this patch tested?
New & modified test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116567008
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ *
+ * @see [[StateStoreWriter.produceOutputWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ // contains the association for batchId -> (stateful operator ID -> input watermark)
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ /**
+ * Retrieve the available input watermarks for specific node in the plan. Every child will
+ * produce an output watermark, which we capture as input watermark. If the child provides
+ * default watermark value (no watermark info), it is excluded.
+ */
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, originWatermark)
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ inputWatermarks.min
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ DEFAULT_WATERMARK_MS
+ }
+
+ val outputWatermarkMs = node.produceOutputWatermark(finalInputWatermarkMs)
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ val minCurrInputWatermarkMs = inputWatermarks.min
+ minCurrInputWatermarkMs
+ } else {
+ DEFAULT_WATERMARK_MS
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ // In current Spark's logic, event time watermark cannot go down to negative. So even there is
+ // no input watermark for operator, the final input watermark for operator should be 0L.
+ inputWatermarks(batchId).get(stateOpId) match {
+ case Some(wm) => Math.max(0L, wm)
+ case None => throw new IllegalStateException(s"Watermark for batch ID $batchId and " +
+ s"stateOpId $stateOpId is not yet set!")
+ }
+ }
+ }
+
+ override def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId - 1, stateOpId)
+
+ override def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ inputWatermarks.remove(currKey)
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+object WatermarkPropagator {
+ val DEFAULT_WATERMARK_MS = -1L
Review Comment:
Can this be an optional instead of negative? The logic to filter these operators with no watermarks may be better. Could there be a situation in the future where a user would like to set -1 as the output watermark for a stateful UDF to signify no advancement of watermarks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115389870
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
> Many stateful operators including streaming aggregation only takes columns in grouping keys into consideration when figuring out event time column. The test case uses count(*) which feels like all columns are being used, but columns in non-grouping key are not effective on determining event time columns.
Could you point to the code where this decision is made? Trying to see why it matters since OutputWatermark for join() and groupBy() don't change even if we pick one or the other.
I.e. what would go wrong if we picked the wrong event-time column due to a bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1102329153
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -113,20 +114,28 @@ class IncrementalExecution(
numStateStores)
}
- // Watermarks to use for late record filtering and state eviction in stateful operators.
- // Using the previous watermark for late record filtering is a Spark behavior change so we allow
- // this to be disabled.
- val eventTimeWatermarkForEviction = offsetSeqMetadata.batchWatermarkMs
- val eventTimeWatermarkForLateEvents =
- if (sparkSession.conf.get(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE)) {
- prevOffsetSeqMetadata.getOrElse(offsetSeqMetadata).batchWatermarkMs
- } else {
- eventTimeWatermarkForEviction
- }
-
/** Locates save/restore pairs surrounding aggregation. */
- val state = new Rule[SparkPlan] {
+ val shufflePartitionsRule = new Rule[SparkPlan] {
+ override def apply(plan: SparkPlan): SparkPlan = plan transform {
+ // NOTE: we should include all aggregate execs here which are used in streaming aggregations
+ case a: SortAggregateExec if a.isStreaming =>
Review Comment:
Is it possible to combine the case matching ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115405490
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
I've been thinking through this again... I might be probably overlooked.
My thought was that if there are two event time columns, the state row should wait for the watermark to pass by two values (W > et1 && W > et2) so that further input won't match with the row. But it seems like the condition is OR, not AND. If either W > et1 or W > et2 happens, further input won't match with the row, as W is the lower bound of both columns.
I probably need to revisit this tomorrow again, or postpone to the next JIRA ticket. If you can help me finding some test/use case which shows expected behavior (and probably an edge-case), that should be awesome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119260122
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ *
+ * @see [[StateStoreWriter.produceOutputWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ // contains the association for batchId -> (stateful operator ID -> input watermark)
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Option[Long]]] =
+ mutable.Map[Long, Map[Long, Option[Long]]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ /**
+ * Retrieve the available input watermarks for specific node in the plan. Every child will
+ * produce an output watermark, which we capture as input watermark. If the child provides
+ * default watermark value (no watermark info), it is excluded.
+ */
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Option[Long]]): Seq[Long] = {
+ node.children.flatMap { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ // Since we use flatMap here, this will exclude children from watermark calculation
+ // which don't have watermark information.
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Option[Long]]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Option[Long]]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, Some(originWatermark))
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ Some(inputWatermarks.min)
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ None
+ }
+
+ val outputWatermarkMs = finalInputWatermarkMs.flatMap { wm =>
+ node.produceOutputWatermark(wm)
+ }
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ Some(inputWatermarks.min)
+ } else {
+ None
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ // In current Spark's logic, event time watermark cannot go down to negative. So even there is
+ // no input watermark for operator, the final input watermark for operator should be 0L.
+ inputWatermarks(batchId).get(stateOpId) match {
+ case Some(wmOpt) => Math.max(wmOpt.getOrElse(0L), 0L)
+ case None => throw new IllegalStateException(s"Watermark for batch ID $batchId and " +
+ s"stateOpId $stateOpId is not yet set!")
+ }
+ }
+ }
+
+ override def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId - 1, stateOpId)
Review Comment:
nit. Can you add a comment on why its "batch - 1"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1432559295
Thanks for reviewing, would you all mind if I ask for another round of review? Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109084691
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -271,8 +343,12 @@ class IncrementalExecution(
* updated metadata.
Review Comment:
The existing explanation is still valid. If we want to be verbose then I'll explain the new change into comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1124328051
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala:
##########
@@ -212,6 +212,8 @@ class MicroBatchExecution(
logInfo(s"Query $prettyIdString was stopped")
}
+ private val watermarkPropagator = WatermarkPropagator(sparkSession.sessionState.conf)
Review Comment:
Whenever we can provide the origin watermark, it can deterministically simulate the watermark propagation. We can do this deterministically from the global watermark info in the offset metadata log/commit log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1102397086
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -1845,9 +1845,7 @@ Some of them are as follows.
for more details.
- Chaining multiple stateful operations on streaming Datasets is not supported with Update and Complete mode.
- - In addition, below operations followed by other stateful operation is not supported in Append mode.
- - stream-stream time interval join (inner/outer)
- - flatMapGroupsWithState
+ - In addition, (flat)MapGroupsWithState operation followed by other stateful operation is not supported in Append mode.
Review Comment:
I thought () was sufficient - will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109081319
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,52 +219,121 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
Review Comment:
That should go to the explanation of WatermarkPropagator. This rule does not need to know about how propagation works. It just calls propagate as WatermarkPropagator requires it before retrieving the input watermark for certain operator, and then calls getInputWatermarkForEviction / getInputWatermarkForLateEvents.
The main reason about not explaining the details of watermark propagation here is that we pick different propagator type based on the config. If you enable compatibility flag, the propagator instance actually does nothing and gives the same watermark value for all calls for getInputWatermarkForEviction / getInputWatermarkForLateEvents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109074676
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Long = inputWatermarkMs
Review Comment:
Currently only join operator is over-riding this, correct ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107138992
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
Review Comment:
I guess I had a code for comparison of class via isInstance. Will change to object if I no longer use that (or will change to `NoOpWatermarkPropagator$`).
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
Review Comment:
I guess I had a code for comparison of class via isInstance. Will change to object if I no longer use that (or will change to `NoOpWatermarkPropagator$` on comparison).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107135448
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,52 +219,119 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
+ private def simulateWatermarkPropagation(plan: SparkPlan): Unit = {
+ val watermarkForPrevBatch = prevOffsetSeqMetadata.map(_.batchWatermarkMs).getOrElse(0L)
+ val watermarkForCurrBatch = offsetSeqMetadata.batchWatermarkMs
+
+ // This is to simulate watermark propagation for late events.
+ watermarkPropagator.propagate(currentBatchId - 1, plan, watermarkForPrevBatch)
+ // This is to simulate watermark propagation for eviction.
+ watermarkPropagator.propagate(currentBatchId, plan, watermarkForCurrBatch)
+ }
+
+ private def inputWatermarkForLateEvents(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForLateEvents(currentBatchId,
+ stateInfo.operatorId))
+ }
+
+ private def inputWatermarkForEviction(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForEviction(currentBatchId, stateInfo.operatorId))
+ }
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+ simulateWatermarkPropagation(plan)
Review Comment:
This should be done against entire plan, not a single node. I understand we would like to simulate watermark propagation only when there is at least one stateful operator, but the cost of simulation shouldn't be noticeable. Spark already applies a bunch of rules against logical plan which traverse the tree. This sounds to me as micro-optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119723950
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
And... even though the semantic is simply wrong, 0-5 window will be buffered in 3-6 window range, even for this implementation. Let's revisit your example:
batch 1: wm1 = 3 wm2 = 3. wm2 may try to evict the buffered window but nothing is there.
and data 4, 5, 6, 7 has been ingested.
batch 2: wm1 = 7 wm2 = 7, 0-5 window is produced from op1 which is NOT a late record for wm2 (it was 3), hence processed, and then wm2 for eviction (7) is applied and 3-6 window is produced from op2.
Does this address your concern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116218223
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
https://github.com/apache/spark/pull/39931/files#diff-da6ad0bc819dce994a16436fa0797bfc8484644b475227a04c2a3eb5927515f7R352-R386
^^^ The existing logic only handles one event time column. We create an expression based on figured out column in caller site.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1114899043
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
Review Comment:
Yeah that's why it's outdated. I changed it. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109097952
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
Review Comment:
Yup, good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119310533
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
Review Comment:
So what is the relationship between this and the WatermarkTracker? The WatermarkTracker is still responsible to determining the global watermark? There is also a map in WatermarkTracker i.e. "operatorToWatermarkMap" that is used to track the watermark on a per operator basis and is used to determine the global watermark. Wouldn't the calculation for global watermark change if we are changing how watermarks for operators are calculated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119407026
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
Your example is not consistent. If you want to advance watermark "after" you produce output, both 5 minutes and 10 minutes window operator watermark should be 0.
Even we provide the output watermark as same as input watermark, it does not violate the definition of watermark, lower bound of the "output". Advancing watermark as 3 for 5 minutes window does not break the logic. Further output will be at least 5, not less than 3, hence the previous advancement of watermark is never violated. Current watermark logic has been working like this way. This is applied for same on 10 minutes window.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119688075
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
So given the current logic:
wm1 = input watermark for 5 minute window
wm2 = input watermark for 10 minute window
Batch 0: wm1 = 0 wm2 = 0
batch 1: wm1 = 3 wm2 = 3
While this strategy may work in the current implementation and for most use cases, is it comprehensive? This works in situations where you have have windowed operators with increasing window sizes but as you pointed out what if that is not true. Like in the example you provided source -> 5 minutes window -> 3 minutes window, the 3 minute window would output a result before the 5 minute window would which is incorrect. The example you provided if the 5 minute window operator outputted a window of 0-5, this window record should be buffered as an entry in 3-6 window range for the 3 minute window. I don't have a specific use case in mind of why someone would do this but why shouldn't it be possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119723950
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
And... even though the semantic is simply wrong, 0-5 window will be buffered in 3-6 window range, even for this implementation. Let's revisit your example:
batch 1: wm1 = 3 wm2 = 3. wm2 may try to evict the buffered window but nothing is there.
and data 4, 5, 6, 7 has been ingested.
batch 2: wm1 = 7 wm2 = 7, 0-5 window is produced from op1 which is NOT a late record for wm2 (it was 3), hence processed, and then wm2 for eviction (7) is applied and 3-6 window is produced from op2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116477889
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
I see more and more places which simply pick up the first occurrence of event time column. (Logic in stream-stream join also assumes there is only one event time column which I missed. Will add assertion.)
That said, considering multiple event time columns for all stateful operators needs non-trivial reasoning. Let's leave this as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
URL: https://github.com/apache/spark/pull/39931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107149084
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,20 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ */
+ def produceWatermark(inputWatermarkMs: Long): Long
Review Comment:
Sounds better!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1420943426
cc. @zsxwing @viirya @jerrypeng @rangadi Please take a look. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1122433137
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
Review Comment:
Yes. It's actually obvious as it should be guaranteed by the definition of watermark, but it could be a bit confusing because we "conceptually" advance the watermark "after" the operator processes the input. That's why we start to use two watermark values for a single microbatch. It should be also explained as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119290314
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -324,6 +349,41 @@ object WatermarkSupport {
}
Some(evictionExpression)
}
+
+ /**
+ * Find the column which is marked as "event time" column.
+ *
+ * If there are multiple event time columns in given column list, the behavior depends on the
+ * parameter `useFirstOccurrence`. If it's set to true, the first occurred column will be
+ * returned. If not, this method will throw an AnalysisException as it is not allowed to have
+ * multiple event time columns.
+ */
+ def findEventTimeColumn(
+ attrs: Seq[Attribute],
+ useFirstOccurrence: Boolean): Option[Attribute] = {
Review Comment:
The name for this variable is kind of confusing. Can we rename it something like "allowMultipleEventTimeColumns"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107145581
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, originWatermark)
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ inputWatermarks.min
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ DEFAULT_WATERMARK_MS
+ }
+
+ val outputWatermarkMs = node.produceWatermark(finalInputWatermarkMs)
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ val minCurrInputWatermarkMs = inputWatermarks.min
+ minCurrInputWatermarkMs
+ } else {
+ DEFAULT_WATERMARK_MS
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
Review Comment:
populate() is called for entire physical plan. Then others will be clarified automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109081319
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,52 +219,121 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
Review Comment:
That should go to the explanation of WatermarkPropagator. This rule does not need to know about how propagation works. It just calls propagate as WatermarkPropagator requires it before retrieving the input watermark for certain operator, and then calls getInputWatermarkForEviction / getInputWatermarkForLateEvents.
The main reason about not explaining the details of watermark propagation here is that we pick different propagator type based on the config. If you enable compatibility flag, the propagator instance actually does nothing about propagation, and gives the same watermark value for all calls for getInputWatermarkForEviction / getInputWatermarkForLateEvents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1442381164
I'll file a JIRA ticket as a follow-up. Thanks for the helping hands!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1441405213
> I'm afraid of adding a test having a non-trivial number of stateful operators, as you've already seen, reasoning about the output (in other words, code comment to explain the behavior) would grow linearly based on the number of stateful operators. Let me give a try with 3.
I think it is ok skip the long comment to explain the final output for this test. A straight forward pipeline with many stateful stage can really stress the code. We could even build the DAG with a for-loop :-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116632454
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Long = inputWatermarkMs
Review Comment:
We could probably make a change along with https://github.com/apache/spark/pull/39931#discussion_r1116567008. Not a big deal.
Just want to call out that optional watermark value is only valid on watermark propagator - current stateful operator implementation assumes there is always a valid watermark value. If watermark spec is not defined, it's 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116629491
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala:
##########
@@ -699,4 +703,18 @@ case class StreamingSymmetricHashJoinExec(
} else {
Nil
}
+
+ // This operator will evict based on the state watermark on both side of inputs; we would like
+ // to let users leverage both sides of event time column for output of join, so the watermark
+ // must be lower bound of both sides of event time column. The lower bound of event time column
+ // for each side is determined by state watermark, hence we take a minimum of (left state
+ // watermark, right state watermark, input watermark) to decide the output watermark.
+ override def produceOutputWatermark(inputWatermarkMs: Long): Long = {
Review Comment:
We provide a single input watermark value via min input watermark of all upstreams. We don't try to be very tight on this - we simply use global watermark as origin watermark rather than individual watermark node as origin watermark, which is already very loose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119420668
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
The only case this could become wrong is something like source -> 5 minutes window -> 3 minutes window, but can someone ever infer the right behavior and output for this case? Is it semantically making sense? I don't think so, so I just excluded the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107150829
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -809,6 +860,10 @@ case class SessionWindowStateStoreSaveExec(
newNumRowsUpdated = stateOpProgress.numRowsUpdated,
newNumRowsDroppedByWatermark = numRowsDroppedByWatermark)
}
+
+ // This operator will evict based on min input watermark and ensure it will be minimum of
+ // the event time value for the output so far (including output from eviction).
+ override def produceWatermark(inputWatermarkMs: Long): Long = inputWatermarkMs
Review Comment:
Yeah... mostly this method will be only implemented via built-in operators and most operators don't need to adjust the watermark. Sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107142832
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
Review Comment:
Yes, the physical plan is a tree and leaves are source (or equivalent one) and root is mostly sink (or final operator).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107137633
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
Review Comment:
Now I get what you've said in previous comment. This method receives the entire plan, not each node. The method will traverse the tree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1421903360
Test failure only happened in k8s integration which is not related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1102327571
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -1845,9 +1845,7 @@ Some of them are as follows.
for more details.
- Chaining multiple stateful operations on streaming Datasets is not supported with Update and Complete mode.
- - In addition, below operations followed by other stateful operation is not supported in Append mode.
- - stream-stream time interval join (inner/outer)
- - flatMapGroupsWithState
+ - In addition, (flat)MapGroupsWithState operation followed by other stateful operation is not supported in Append mode.
Review Comment:
This includes mapGroupsWithState too right ? should we just call them out separately ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1124322044
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,63 +222,140 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
+ private def simulateWatermarkPropagation(plan: SparkPlan): Unit = {
+ val watermarkForPrevBatch = prevOffsetSeqMetadata.map(_.batchWatermarkMs).getOrElse(0L)
+ val watermarkForCurrBatch = offsetSeqMetadata.batchWatermarkMs
+
+ // This is to simulate watermark propagation for late events.
+ watermarkPropagator.propagate(currentBatchId - 1, plan, watermarkForPrevBatch)
+ // This is to simulate watermark propagation for eviction.
+ watermarkPropagator.propagate(currentBatchId, plan, watermarkForCurrBatch)
+ }
+
+ private def inputWatermarkForLateEvents(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForLateEvents(currentBatchId,
+ stateInfo.operatorId))
+ }
+
+ private def inputWatermarkForEviction(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForEviction(currentBatchId, stateInfo.operatorId))
+ }
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+ simulateWatermarkPropagation(plan)
+ plan transform {
+ case s: StateStoreSaveExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: SessionWindowStateStoreSaveExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: SessionWindowStateStoreRestoreExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: StreamingDeduplicateExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case m: FlatMapGroupsWithStateExec if m.stateInfo.isDefined =>
+ m.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(m.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(m.stateInfo.get)
+ )
+
+ case m: FlatMapGroupsInPandasWithStateExec if m.stateInfo.isDefined =>
+ m.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(m.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(m.stateInfo.get)
+ )
+
+ case j: StreamingSymmetricHashJoinExec =>
+ val iwLateEvents = inputWatermarkForLateEvents(j.stateInfo.get)
+ val iwEviction = inputWatermarkForEviction(j.stateInfo.get)
+ j.copy(
+ eventTimeWatermarkForLateEvents = iwLateEvents,
+ eventTimeWatermarkForEviction = iwEviction,
+ stateWatermarkPredicates =
+ StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
+ j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
+ iwEviction, !allowMultipleStatefulOperators)
+ )
+ }
}
}
- override def preparations: Seq[Rule[SparkPlan]] = state +: super.preparations
+ override def preparations: Seq[Rule[SparkPlan]] = Seq(
+ shufflePartitionsRule,
Review Comment:
The actual reason is that issuing state op ID needs to be performed "before" simulate propagation. (That said, it needs to traverse the entire tree at least 2 times.) After that, I feel like rule for issuing deals with so many things by itself compared to the propagation, hence refactor out to similar level of rules. Yeah you're right some of them are for refactoring and it may trigger 2 more passes.
I'll see whether we can apply pruning with the node pattern, and compose them into two if there is no good way to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1124464792
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,63 +222,140 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
+ private def simulateWatermarkPropagation(plan: SparkPlan): Unit = {
+ val watermarkForPrevBatch = prevOffsetSeqMetadata.map(_.batchWatermarkMs).getOrElse(0L)
+ val watermarkForCurrBatch = offsetSeqMetadata.batchWatermarkMs
+
+ // This is to simulate watermark propagation for late events.
+ watermarkPropagator.propagate(currentBatchId - 1, plan, watermarkForPrevBatch)
+ // This is to simulate watermark propagation for eviction.
+ watermarkPropagator.propagate(currentBatchId, plan, watermarkForCurrBatch)
+ }
+
+ private def inputWatermarkForLateEvents(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForLateEvents(currentBatchId,
+ stateInfo.operatorId))
+ }
+
+ private def inputWatermarkForEviction(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForEviction(currentBatchId, stateInfo.operatorId))
+ }
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+ simulateWatermarkPropagation(plan)
+ plan transform {
+ case s: StateStoreSaveExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: SessionWindowStateStoreSaveExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: SessionWindowStateStoreRestoreExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: StreamingDeduplicateExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case m: FlatMapGroupsWithStateExec if m.stateInfo.isDefined =>
+ m.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(m.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(m.stateInfo.get)
+ )
+
+ case m: FlatMapGroupsInPandasWithStateExec if m.stateInfo.isDefined =>
+ m.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(m.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(m.stateInfo.get)
+ )
+
+ case j: StreamingSymmetricHashJoinExec =>
+ val iwLateEvents = inputWatermarkForLateEvents(j.stateInfo.get)
+ val iwEviction = inputWatermarkForEviction(j.stateInfo.get)
+ j.copy(
+ eventTimeWatermarkForLateEvents = iwLateEvents,
+ eventTimeWatermarkForEviction = iwEviction,
+ stateWatermarkPredicates =
+ StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
+ j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
+ iwEviction, !allowMultipleStatefulOperators)
+ )
+ }
}
}
- override def preparations: Seq[Rule[SparkPlan]] = state +: super.preparations
+ override def preparations: Seq[Rule[SparkPlan]] = Seq(
+ shufflePartitionsRule,
Review Comment:
I just addressed this. Please refer to the change of [c39b642](https://github.com/apache/spark/pull/39931/commits/c39b64284b1298c358fa559ec1b91d0d2f7739a5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115381728
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
`groupBy` here gets two event timestamp columns too, right, similar to dedup(). What is the difference?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1441393531
Many stateful operators including streaming aggregation only takes columns in grouping keys into consideration when figuring out event time column. The test case uses `count(*)` which feels like all columns are being used, but columns in non-grouping key are not effective on determining event time columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1420952309
IMHO, the code change and workload this enables don't seem to be complicated warranting SPIP. But I'm definitely open to go with SPIP if any committer thinks this should.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107143368
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
Review Comment:
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107190717
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,52 +219,119 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
+ private def simulateWatermarkPropagation(plan: SparkPlan): Unit = {
+ val watermarkForPrevBatch = prevOffsetSeqMetadata.map(_.batchWatermarkMs).getOrElse(0L)
+ val watermarkForCurrBatch = offsetSeqMetadata.batchWatermarkMs
+
+ // This is to simulate watermark propagation for late events.
+ watermarkPropagator.propagate(currentBatchId - 1, plan, watermarkForPrevBatch)
+ // This is to simulate watermark propagation for eviction.
+ watermarkPropagator.propagate(currentBatchId, plan, watermarkForCurrBatch)
+ }
+
+ private def inputWatermarkForLateEvents(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForLateEvents(currentBatchId,
+ stateInfo.operatorId))
+ }
+
+ private def inputWatermarkForEviction(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForEviction(currentBatchId, stateInfo.operatorId))
+ }
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+ simulateWatermarkPropagation(plan)
Review Comment:
(Looks like there is misunderstanding/miscommunication.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109085186
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
Review Comment:
So this is the default implementation currently ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109071490
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -271,8 +343,12 @@ class IncrementalExecution(
* updated metadata.
Review Comment:
Should we update this comment now ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1457520455
Thanks all for quite huge efforts on reviewing this complicated change! The implementation got better with the review comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1124325417
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceOutputWatermark(input watermark)`.
+ *
+ * @see [[StateStoreWriter.produceOutputWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ // contains the association for batchId -> (stateful operator ID -> input watermark)
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Option[Long]]] =
+ mutable.Map[Long, Map[Long, Option[Long]]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ /**
+ * Retrieve the available input watermarks for specific node in the plan. Every child will
+ * produce an output watermark, which we capture as input watermark. If the child provides
+ * default watermark value (no watermark info), it is excluded.
+ */
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Option[Long]]): Seq[Long] = {
+ node.children.flatMap { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ // Since we use flatMap here, this will exclude children from watermark calculation
+ // which don't have watermark information.
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Option[Long]]()
Review Comment:
Unless the batch side has a crazily huge and complicated plan, it should not be a big deal. I would think this is a matter of "sub" millisecond.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1124328051
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala:
##########
@@ -212,6 +212,8 @@ class MicroBatchExecution(
logInfo(s"Query $prettyIdString was stopped")
}
+ private val watermarkPropagator = WatermarkPropagator(sparkSession.sessionState.conf)
Review Comment:
Whenever we can provide the origin watermark, it can deterministically simulate the watermark propagation. (Assume there is no change on physical plan of the query of course.) We can do this deterministically from the global watermark info in the offset metadata log/commit log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1441448096
I'm sorry, I'm afraid I still don't get the goal of the test you're proposing. Do you want to see a test case running a query having a bunch of stateful operators without any correctness check, or a test case running a query having reasonable number of stateful operators with correctness check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116519383
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
https://issues.apache.org/jira/browse/SPARK-42549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115384093
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
> For the test case of join + groupBy, we picked one event time column for time window and discarded other one.
Moving the discussion to the comment thread here.
Could you expand the rationale here? What is the watermark here? Who decides which timestamp is picked? What is the watermark based on (or is it independent of event timestamp picked?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1441384161
> How about adding a simple test with 4 or 5 aggregations :)?
I'm afraid of adding a test having a non-trivial number of stateful operators, as you've already seen, reasoning about the output would grow linearly based on the number of stateful operators. Let me give a try with 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1114957687
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala:
##########
@@ -548,17 +549,131 @@ class EventTimeWatermarkSuite extends StreamTest with BeforeAndAfter with Matche
assert(e.getMessage contains "should not be negative.")
}
- test("the new watermark should override the old one") {
- val df = MemoryStream[(Long, Long)].toDF()
+ private def buildTestQueryForOverridingWatermark(): (MemoryStream[(Long, Long)], DataFrame) = {
+ val input = MemoryStream[(Long, Long)]
+ val df = input.toDF()
.withColumn("first", timestamp_seconds($"_1"))
.withColumn("second", timestamp_seconds($"_2"))
.withWatermark("first", "1 minute")
+ .select("*")
.withWatermark("second", "2 minutes")
+ .groupBy(window($"second", "1 minute"))
+ .count()
- val eventTimeColumns = df.logicalPlan.output
- .filter(_.metadata.contains(EventTimeWatermark.delayKey))
- assert(eventTimeColumns.size === 1)
- assert(eventTimeColumns(0).name === "second")
+ (input, df)
+ }
+
+ test("overriding watermark should not be allowed by default") {
+ val (input, df) = buildTestQueryForOverridingWatermark()
+ testStream(df)(
+ AddData(input, (100L, 200L)),
+ ExpectFailure[AnalysisException](assertFailure = exc => {
+ assert(exc.getMessage.contains("Redefining watermark is disallowed."))
+ assert(exc.getMessage.contains(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key))
+ })
+ )
+ }
+
+ test("overriding watermark should not fail in compatibility mode") {
+ withSQLConf(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key -> "false") {
+ val (input, df) = buildTestQueryForOverridingWatermark()
+ testStream(df)(
+ AddData(input, (100L, 200L)),
+ CheckAnswer(),
+ Execute { query =>
+ val lastExecution = query.lastExecution
+ val aggSaveOperator = lastExecution.executedPlan.collect {
+ case j: StateStoreSaveExec => j
+ }.head
+
+ // - watermark from first definition = 100 - 60 = 40
+ // - watermark from second definition = 200 - 120 = 80
+ // - global watermark = min(40, 60) = 40
+ //
+ // As we see the result, even though we override the watermark definition, the old
+ // definition of watermark still plays to calculate global watermark.
+ //
+ // This is conceptually the right behavior. For operators after the first watermark
+ // definition, the column named "first" is considered as event time column, and for
+ // operators after the second watermark definition, the column named "second" is
+ // considered as event time column. The correct "single" value of watermark satisfying
+ // all operators should be lower bound of both columns "first" and "second".
+ //
+ // That said, this easily leads to incorrect definition - e.g. re-define watermark
+ // against the output of streaming aggregation for append mode. The global watermark
+ // cannot advance. This is the reason we don't allow re-define watermark in new behavior.
+ val expectedWatermarkMs = 40 * 1000
+
+ assert(aggSaveOperator.eventTimeWatermarkForLateEvents === Some(expectedWatermarkMs))
+ assert(aggSaveOperator.eventTimeWatermarkForEviction === Some(expectedWatermarkMs))
+
+ val eventTimeCols = aggSaveOperator.keyExpressions.filter(
+ _.metadata.contains(EventTimeWatermark.delayKey))
+ assert(eventTimeCols.size === 1)
+ assert(eventTimeCols.head.name === "window")
+ // 2 minutes delay threshold
+ assert(eventTimeCols.head.metadata.getLong(EventTimeWatermark.delayKey) === 120 * 1000)
+ }
+ )
+ }
+ }
+
+ private def buildTestQueryForMultiEventTimeColumns()
+ : (MemoryStream[(String, Long)], MemoryStream[(String, Long)], DataFrame) = {
+ val input1 = MemoryStream[(String, Long)]
+ val input2 = MemoryStream[(String, Long)]
+ val df1 = input1.toDF()
+ .selectExpr("_1 AS id1", "timestamp_seconds(_2) AS ts1")
+ .withWatermark("ts1", "1 minute")
+
+ val df2 = input2.toDF()
+ .selectExpr("_1 AS id2", "timestamp_seconds(_2) AS ts2")
+ .withWatermark("ts2", "2 minutes")
+
+ val joined = df1.join(df2, expr("id1 = id2 AND ts1 = ts2 + INTERVAL 10 SECONDS"), "inner")
+ .selectExpr("id1", "ts1", "ts2")
+ // the output of join contains both ts1 and ts2
+ val dedup = joined.dropDuplicates()
+ .selectExpr("id1", "CAST(ts1 AS LONG) AS ts1", "CAST(ts2 AS LONG) AS ts2")
+
+ (input1, input2, dedup)
+ }
+
+ test("multiple event time columns in an input DataFrame for stateful operator is " +
Review Comment:
How do we make this query work? Looks like correct query.
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
Review Comment:
Minor: I think `.toDF("id1", "timestamp1")` is equivalent to this.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
Review Comment:
Yes. Just to help the readers.
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
How is this different from `join() followed deDup()` test in EventTimeWatermarkSuite.scala? The latter was not allowed.
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
+ .agg(count("*").as("cnt"))
+ .selectExpr("CAST(window.start AS STRING) AS window_start",
+ "CAST(window.end AS STRING) AS window_end", "cnt")
+
+ // for ease of verification, we change the session timezone to UTC
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ testStream(agg)(
+ MultiAddData(
+ (input1, Seq(
+ ("1", Timestamp.valueOf("2023-01-01 01:00:10")),
+ ("2", Timestamp.valueOf("2023-01-01 01:00:30")))
+ ),
+ (input2, Seq(
+ ("1", Timestamp.valueOf("2023-01-01 01:00:20"))))
+ ),
+
+ // < data batch >
+ // global watermark (0, 0)
+ // op1 (join)
+ // -- IW (0, 0)
+ // -- OW 0
+ // -- left state
+ // ("1", "2023-01-01 01:00:10", matched=true)
+ // ("1", "2023-01-01 01:00:30", matched=false)
+ // -- right state
+ // ("1", "2023-01-01 01:00:20")
+ // -- result
+ // ("1", "2023-01-01 01:00:10", "1", "2023-01-01 01:00:20")
+ // op2 (aggregation)
+ // -- IW (0, 0)
+ // -- OW 0
+ // -- state row
+ // ("2023-01-01 01:00:00", "2023-01-01 01:10:00", 1)
+ // -- result
+ // None
+
+ // -- watermark calculation
+ // watermark in left input: 2023-01-01 01:00:30
+ // watermark in right input: 2023-01-01 01:00:20
+ // origin watermark: 2023-01-01 01:00:20
+
+ // < no-data batch >
+ // global watermark (0, 2023-01-01 01:00:20)
+ // op1 (join)
+ // -- IW (0, 2023-01-01 01:00:20)
+ // -- OW 2023-01-01 00:00:19.999999
+ // -- left state
+ // ("1", "2023-01-01 01:00:10", matched=true)
+ // ("1", "2023-01-01 01:00:30", matched=false)
+ // -- right state
+ // ("1", "2023-01-01 01:00:20")
+ // -- result
+ // None
+ // op2 (aggregation)
+ // -- IW (0, 2023-01-01 00:00:19.999999)
+ // -- OW 2023-01-01 00:00:19.999999
+ // -- state row
+ // ("2023-01-01 01:00:00", "2023-01-01 01:10:00", 1)
+ // -- result
+ // None
+ CheckAnswer(),
+
+ Execute { query =>
+ val lastExecution = query.lastExecution
+ val joinOperator = lastExecution.executedPlan.collect {
+ case j: StreamingSymmetricHashJoinExec => j
+ }.head
+ val aggSaveOperator = lastExecution.executedPlan.collect {
+ case j: StateStoreSaveExec => j
+ }.head
+
+ assert(joinOperator.eventTimeWatermarkForLateEvents === Some(0))
+ assert(joinOperator.eventTimeWatermarkForEviction ===
+ Some(Timestamp.valueOf("2023-01-01 01:00:20").getTime))
+
+ assert(aggSaveOperator.eventTimeWatermarkForLateEvents === Some(0))
+ assert(aggSaveOperator.eventTimeWatermarkForEviction ===
+ Some(Timestamp.valueOf("2023-01-01 00:00:20").getTime - 1))
+ },
+
+ MultiAddData(
+ (input1, Seq(("5", Timestamp.valueOf("2023-01-01 01:15:00")))),
+ (input2, Seq(("6", Timestamp.valueOf("2023-01-01 01:15:00"))))
+ ),
+
+ // < data batch >
Review Comment:
Thanks for these detailed comments in the tests.
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
Review Comment:
This is a detailed test with great comments inline. It will nice to give a short overview of this test does at the top here.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, originWatermark)
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ inputWatermarks.min
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ DEFAULT_WATERMARK_MS
+ }
+
+ val outputWatermarkMs = node.produceWatermark(finalInputWatermarkMs)
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ val minCurrInputWatermarkMs = inputWatermarks.min
+ minCurrInputWatermarkMs
+ } else {
+ DEFAULT_WATERMARK_MS
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
Review Comment:
[For understanding]: Where is called? want to see if we can figure out how it is only called only once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1442406888
https://issues.apache.org/jira/browse/SPARK-42540 for follow-up task to add longer chain of stateful operators as new test case(s).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1102396588
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -113,20 +114,28 @@ class IncrementalExecution(
numStateStores)
}
- // Watermarks to use for late record filtering and state eviction in stateful operators.
- // Using the previous watermark for late record filtering is a Spark behavior change so we allow
- // this to be disabled.
- val eventTimeWatermarkForEviction = offsetSeqMetadata.batchWatermarkMs
- val eventTimeWatermarkForLateEvents =
- if (sparkSession.conf.get(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE)) {
- prevOffsetSeqMetadata.getOrElse(offsetSeqMetadata).batchWatermarkMs
- } else {
- eventTimeWatermarkForEviction
- }
-
/** Locates save/restore pairs surrounding aggregation. */
- val state = new Rule[SparkPlan] {
+ val shufflePartitionsRule = new Rule[SparkPlan] {
+ override def apply(plan: SparkPlan): SparkPlan = plan transform {
+ // NOTE: we should include all aggregate execs here which are used in streaming aggregations
+ case a: SortAggregateExec if a.isStreaming =>
Review Comment:
The amount of change would be similar - I see one way to do this, `def withNewNumShufflePartitions(numStateStores: Long): BaseAggregateExec` but this may need to be implemented per class. All logical/physical nodes are case class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] chaoqin-li1123 commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "chaoqin-li1123 (via GitHub)" <gi...@apache.org>.
chaoqin-li1123 commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1104913618
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
Review Comment:
It seems that IncrementalExecution planning and purging run sequentially, so there is no need to synchronize the write. Will there be any future use case that access this cache interface concurrently? Should we document this is not thread-safe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107148761
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, originWatermark)
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ inputWatermarks.min
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ DEFAULT_WATERMARK_MS
+ }
+
+ val outputWatermarkMs = node.produceWatermark(finalInputWatermarkMs)
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ val minCurrInputWatermarkMs = inputWatermarks.min
+ minCurrInputWatermarkMs
+ } else {
+ DEFAULT_WATERMARK_MS
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ // In current Spark's logic, event time watermark cannot go down to negative. So even there is
+ // no input watermark for operator, the final input watermark for operator should be 0L.
+ val opWatermark = inputWatermarks(batchId).get(stateOpId)
Review Comment:
That sounds fair for this case. Do you see other places? It would be really helpful if you can help commenting like `ditto`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115349019
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, originWatermark)
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ inputWatermarks.min
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ DEFAULT_WATERMARK_MS
+ }
+
+ val outputWatermarkMs = node.produceWatermark(finalInputWatermarkMs)
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ val minCurrInputWatermarkMs = inputWatermarks.min
+ minCurrInputWatermarkMs
+ } else {
+ DEFAULT_WATERMARK_MS
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
Review Comment:
Planning of the streaming query (IncrementalExecution), and checking for no-data batch execution (MicrobatchExecution).
So it's not only called once - we allow this method to be called multiple times. We just leverage cached result if it had been calculated before. We assume that physical plan of the query won't change across microbatches, and we guarantee we give the same origin watermark value for the same batch ID. (Otherwise assertion will crash the query.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115965485
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
May be we can leave a comment and open to ticket to address it as a follow up. But I think it is an important issue as it relates to core of the feature. I will think about it deeper to. My question at high level is "what is the difference that makes this ok for windowed count, but not ok for deDup()"
Could you point to the code where this difference comes from? I will look deeper as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116601274
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala:
##########
@@ -699,4 +703,18 @@ case class StreamingSymmetricHashJoinExec(
} else {
Nil
}
+
+ // This operator will evict based on the state watermark on both side of inputs; we would like
+ // to let users leverage both sides of event time column for output of join, so the watermark
+ // must be lower bound of both sides of event time column. The lower bound of event time column
+ // for each side is determined by state watermark, hence we take a minimum of (left state
+ // watermark, right state watermark, input watermark) to decide the output watermark.
+ override def produceOutputWatermark(inputWatermarkMs: Long): Long = {
Review Comment:
I am a little confused on what the input watermark for join is suppose to be. I would expect there would be two input watermarks from both sides?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116216190
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
dedup() with dropping one of event time column from the list of columns should also work. That's not about which operator it is. It's just that a stateful operator can receive two event time columns, and current logic does not consider the case (or we did it but I don't get it).
Let me think about this once more. We probably wouldn't like to disallow the functionality first and allow it later, hence it sounds to me we need to give a try to avoid disallowing the functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116634066
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ *
+ * @see [[StateStoreWriter.produceOutputWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ // contains the association for batchId -> (stateful operator ID -> input watermark)
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ /**
+ * Retrieve the available input watermarks for specific node in the plan. Every child will
+ * produce an output watermark, which we capture as input watermark. If the child provides
+ * default watermark value (no watermark info), it is excluded.
+ */
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, originWatermark)
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ inputWatermarks.min
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ DEFAULT_WATERMARK_MS
+ }
+
+ val outputWatermarkMs = node.produceOutputWatermark(finalInputWatermarkMs)
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ val minCurrInputWatermarkMs = inputWatermarks.min
+ minCurrInputWatermarkMs
+ } else {
+ DEFAULT_WATERMARK_MS
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ // In current Spark's logic, event time watermark cannot go down to negative. So even there is
+ // no input watermark for operator, the final input watermark for operator should be 0L.
+ inputWatermarks(batchId).get(stateOpId) match {
+ case Some(wm) => Math.max(0L, wm)
+ case None => throw new IllegalStateException(s"Watermark for batch ID $batchId and " +
+ s"stateOpId $stateOpId is not yet set!")
+ }
+ }
+ }
+
+ override def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId - 1, stateOpId)
+
+ override def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ inputWatermarks.remove(currKey)
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+object WatermarkPropagator {
+ val DEFAULT_WATERMARK_MS = -1L
Review Comment:
I think we can make the change anytime when it is needed, as the watermark value per operator is not going to be persisted. But I agree None is clearer than -1 for self-describing the semantic. I'll look into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119277586
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
Why is this the default behavior for stateful operators? Consider a query with two window operators:
source -> 5 minute window - > ten min window
Batch 0:
Input data: 1, 2, 3
5 minute window operator watermark should be 3 after batch 0 but the watermark of the ten minute window should still be 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119409310
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
Review Comment:
The "operators" in operatorToWatermarkMap in WatermarkTracker are watermark nodes. We never had the concept of watermark per operator.
We don't change anything about calculating global watermark. We just start to disallow redefining the watermark which was allowed and root cause of incorrect global watermark value. Once global watermark is calculated from WatermarkTracker, it is considered as origin of watermark for WatermarkPropagator. Nothing should be changed for WatermarkTracker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119692932
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
While we say "representative time" of time window, we still need to reason about the semantic for chained time window aggregation, based on how time windows are "inclusive" in bigger time window.
For example, if we suppose to have a query having 5 mins time window -> 3 mins time window. Are we really sure having any output in 3 mins time window is valid? Where the output of 5 mins time window have to be bound for next 3 mins time window? Why we need to worry about the case which just doesn't make sense at all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1120674821
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
Review Comment:
If the above is correct, I would like to propose an explicit contract written down in the doc comment for this method:
_An operator guarantees that it will not emit record with an event timestamp lower than its output watermark_.
It might be obvious, but I think it is better to explicitly state it. cc: @jerrypeng
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
Review Comment:
Making sure my understanding is correct : As a concrete example: If windowed aggregation (say `count(over 5 minutes`) emits a record X, it's 'Event-timestamp' is 'window-end'. With this, the stateful operator for count() can guarantee that it will not any record in the future with lower timestamp than its output watermark. Is that correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119407197
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ *
+ * @see [[StateStoreWriter.produceOutputWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ // contains the association for batchId -> (stateful operator ID -> input watermark)
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Option[Long]]] =
+ mutable.Map[Long, Map[Long, Option[Long]]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ /**
+ * Retrieve the available input watermarks for specific node in the plan. Every child will
+ * produce an output watermark, which we capture as input watermark. If the child provides
+ * default watermark value (no watermark info), it is excluded.
+ */
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Option[Long]]): Seq[Long] = {
+ node.children.flatMap { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ // Since we use flatMap here, this will exclude children from watermark calculation
+ // which don't have watermark information.
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Option[Long]]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Option[Long]]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, Some(originWatermark))
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ Some(inputWatermarks.min)
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ None
+ }
+
+ val outputWatermarkMs = finalInputWatermarkMs.flatMap { wm =>
+ node.produceOutputWatermark(wm)
+ }
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ Some(inputWatermarks.min)
+ } else {
+ None
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ // In current Spark's logic, event time watermark cannot go down to negative. So even there is
+ // no input watermark for operator, the final input watermark for operator should be 0L.
+ inputWatermarks(batchId).get(stateOpId) match {
+ case Some(wmOpt) => Math.max(wmOpt.getOrElse(0L), 0L)
+ case None => throw new IllegalStateException(s"Watermark for batch ID $batchId and " +
+ s"stateOpId $stateOpId is not yet set!")
+ }
+ }
+ }
+
+ override def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId - 1, stateOpId)
Review Comment:
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1120620417
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
Review Comment:
minor: Fix the name `produceOutputWatermark()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119401204
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala:
##########
@@ -90,19 +90,24 @@ trait FlatMapGroupsWithStateExecBase
override def shortName: String = "flatMapGroupsWithState"
- override def shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean = {
+ override def shouldRunAnotherBatch(newInputWatermark: Long): Boolean = {
timeoutConf match {
case ProcessingTimeTimeout =>
true // Always run batches to process timeouts
case EventTimeTimeout =>
// Process another non-data batch only if the watermark has changed in this executed plan
eventTimeWatermarkForEviction.isDefined &&
- newMetadata.batchWatermarkMs > eventTimeWatermarkForEviction.get
+ newInputWatermark > eventTimeWatermarkForEviction.get
case _ =>
false
}
}
+ // There is no guarantee that any of the column in the output is bound to the watermark. The
+ // user function is quite flexible. Hence Spark does not support the stateful operator(s) after
+ // (flat)MapGroupsWithState.
+ override def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = None
Review Comment:
Spark has no idea about the value of the timestamp for the output. That said, either user should provide the logic to calculate the watermark (I don't think it will work), or the operator has to track the value for the event time (delayed by one batch).
But even before we talk about this, there are lots of issues about the event time with typed API. During conversion between untyped <-> typed Spark loses the metadata in some cases. Furthermore, the output of flatMapGroupsWithState is also typed, which we need to convert back to untyped to apply further stateful operations, hence the risk of losing metadata. In addition, we don't have an ability of specifying new event time in the output for typed API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1120512476
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
@HeartSaVioR thanks for the detailed explanation. Can you add we add this information as a comment in the code so we don't forget the rationale of why we are doing this? Please include some examples if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zsxwing commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "zsxwing (via GitHub)" <gi...@apache.org>.
zsxwing commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1124167428
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,63 +222,140 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
+ private def simulateWatermarkPropagation(plan: SparkPlan): Unit = {
+ val watermarkForPrevBatch = prevOffsetSeqMetadata.map(_.batchWatermarkMs).getOrElse(0L)
+ val watermarkForCurrBatch = offsetSeqMetadata.batchWatermarkMs
+
+ // This is to simulate watermark propagation for late events.
+ watermarkPropagator.propagate(currentBatchId - 1, plan, watermarkForPrevBatch)
+ // This is to simulate watermark propagation for eviction.
+ watermarkPropagator.propagate(currentBatchId, plan, watermarkForCurrBatch)
+ }
+
+ private def inputWatermarkForLateEvents(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForLateEvents(currentBatchId,
+ stateInfo.operatorId))
+ }
+
+ private def inputWatermarkForEviction(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForEviction(currentBatchId, stateInfo.operatorId))
+ }
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+ simulateWatermarkPropagation(plan)
+ plan transform {
+ case s: StateStoreSaveExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: SessionWindowStateStoreSaveExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: SessionWindowStateStoreRestoreExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: StreamingDeduplicateExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case m: FlatMapGroupsWithStateExec if m.stateInfo.isDefined =>
+ m.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(m.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(m.stateInfo.get)
+ )
+
+ case m: FlatMapGroupsInPandasWithStateExec if m.stateInfo.isDefined =>
+ m.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(m.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(m.stateInfo.get)
+ )
+
+ case j: StreamingSymmetricHashJoinExec =>
+ val iwLateEvents = inputWatermarkForLateEvents(j.stateInfo.get)
+ val iwEviction = inputWatermarkForEviction(j.stateInfo.get)
+ j.copy(
+ eventTimeWatermarkForLateEvents = iwLateEvents,
+ eventTimeWatermarkForEviction = iwEviction,
+ stateWatermarkPredicates =
+ StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
+ j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
+ iwEviction, !allowMultipleStatefulOperators)
+ )
+ }
}
}
- override def preparations: Seq[Rule[SparkPlan]] = state +: super.preparations
+ override def preparations: Seq[Rule[SparkPlan]] = Seq(
+ shufflePartitionsRule,
Review Comment:
Is splitting rules just to improve readability? This would traverse the plan tree for 3 more passes. This is probably okay as people usually don't write complicated streaming queries.
Batch queries are using a cheap API [transformUpWithPruning](https://github.com/apache/spark/blob/6874fd1b897680263ffbb3d06f716179aca8e4c566/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala#L557) to traverse tree. Would be great to keep this in mind and consider doing this in future if we need to add more rules.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceOutputWatermark(input watermark)`.
+ *
+ * @see [[StateStoreWriter.produceOutputWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ // contains the association for batchId -> (stateful operator ID -> input watermark)
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Option[Long]]] =
+ mutable.Map[Long, Map[Long, Option[Long]]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ /**
+ * Retrieve the available input watermarks for specific node in the plan. Every child will
+ * produce an output watermark, which we capture as input watermark. If the child provides
+ * default watermark value (no watermark info), it is excluded.
+ */
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Option[Long]]): Seq[Long] = {
+ node.children.flatMap { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ // Since we use flatMap here, this will exclude children from watermark calculation
+ // which don't have watermark information.
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
Review Comment:
Can we skip this heavy work if the query doesn't define a watermark? Most streaming queries are in this case, and don't need to run this in each micro batch.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala:
##########
@@ -212,6 +212,8 @@ class MicroBatchExecution(
logInfo(s"Query $prettyIdString was stopped")
}
+ private val watermarkPropagator = WatermarkPropagator(sparkSession.sessionState.conf)
Review Comment:
How do we recover `WatermarkPropagator` for a query restart?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceOutputWatermark(input watermark)`.
+ *
+ * @see [[StateStoreWriter.produceOutputWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ // contains the association for batchId -> (stateful operator ID -> input watermark)
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Option[Long]]] =
+ mutable.Map[Long, Map[Long, Option[Long]]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ /**
+ * Retrieve the available input watermarks for specific node in the plan. Every child will
+ * produce an output watermark, which we capture as input watermark. If the child provides
+ * default watermark value (no watermark info), it is excluded.
+ */
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Option[Long]]): Seq[Long] = {
+ node.children.flatMap { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ // Since we use flatMap here, this will exclude children from watermark calculation
+ // which don't have watermark information.
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Option[Long]]()
Review Comment:
If we are doing a batch-streaming join, do we still calculate watermarks for the batch side even if they are always empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109070921
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,52 +219,121 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
Review Comment:
Can we add a high level comment explaining how the watermark propagation is working for different stateful operators here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1457521207
Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1124323368
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceOutputWatermark(input watermark)`.
+ *
+ * @see [[StateStoreWriter.produceOutputWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ // contains the association for batchId -> (stateful operator ID -> input watermark)
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Option[Long]]] =
+ mutable.Map[Long, Map[Long, Option[Long]]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ /**
+ * Retrieve the available input watermarks for specific node in the plan. Every child will
+ * produce an output watermark, which we capture as input watermark. If the child provides
+ * default watermark value (no watermark info), it is excluded.
+ */
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Option[Long]]): Seq[Long] = {
+ node.children.flatMap { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ // Since we use flatMap here, this will exclude children from watermark calculation
+ // which don't have watermark information.
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
Review Comment:
Let's skip the simulation when origin watermark is presented to 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1114886982
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
Review Comment:
> class via isInstance
We should be able to check that even with `object`. Let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115352110
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala:
##########
@@ -548,17 +549,131 @@ class EventTimeWatermarkSuite extends StreamTest with BeforeAndAfter with Matche
assert(e.getMessage contains "should not be negative.")
}
- test("the new watermark should override the old one") {
- val df = MemoryStream[(Long, Long)].toDF()
+ private def buildTestQueryForOverridingWatermark(): (MemoryStream[(Long, Long)], DataFrame) = {
+ val input = MemoryStream[(Long, Long)]
+ val df = input.toDF()
.withColumn("first", timestamp_seconds($"_1"))
.withColumn("second", timestamp_seconds($"_2"))
.withWatermark("first", "1 minute")
+ .select("*")
.withWatermark("second", "2 minutes")
+ .groupBy(window($"second", "1 minute"))
+ .count()
- val eventTimeColumns = df.logicalPlan.output
- .filter(_.metadata.contains(EventTimeWatermark.delayKey))
- assert(eventTimeColumns.size === 1)
- assert(eventTimeColumns(0).name === "second")
+ (input, df)
+ }
+
+ test("overriding watermark should not be allowed by default") {
+ val (input, df) = buildTestQueryForOverridingWatermark()
+ testStream(df)(
+ AddData(input, (100L, 200L)),
+ ExpectFailure[AnalysisException](assertFailure = exc => {
+ assert(exc.getMessage.contains("Redefining watermark is disallowed."))
+ assert(exc.getMessage.contains(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key))
+ })
+ )
+ }
+
+ test("overriding watermark should not fail in compatibility mode") {
+ withSQLConf(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key -> "false") {
+ val (input, df) = buildTestQueryForOverridingWatermark()
+ testStream(df)(
+ AddData(input, (100L, 200L)),
+ CheckAnswer(),
+ Execute { query =>
+ val lastExecution = query.lastExecution
+ val aggSaveOperator = lastExecution.executedPlan.collect {
+ case j: StateStoreSaveExec => j
+ }.head
+
+ // - watermark from first definition = 100 - 60 = 40
+ // - watermark from second definition = 200 - 120 = 80
+ // - global watermark = min(40, 60) = 40
+ //
+ // As we see the result, even though we override the watermark definition, the old
+ // definition of watermark still plays to calculate global watermark.
+ //
+ // This is conceptually the right behavior. For operators after the first watermark
+ // definition, the column named "first" is considered as event time column, and for
+ // operators after the second watermark definition, the column named "second" is
+ // considered as event time column. The correct "single" value of watermark satisfying
+ // all operators should be lower bound of both columns "first" and "second".
+ //
+ // That said, this easily leads to incorrect definition - e.g. re-define watermark
+ // against the output of streaming aggregation for append mode. The global watermark
+ // cannot advance. This is the reason we don't allow re-define watermark in new behavior.
+ val expectedWatermarkMs = 40 * 1000
+
+ assert(aggSaveOperator.eventTimeWatermarkForLateEvents === Some(expectedWatermarkMs))
+ assert(aggSaveOperator.eventTimeWatermarkForEviction === Some(expectedWatermarkMs))
+
+ val eventTimeCols = aggSaveOperator.keyExpressions.filter(
+ _.metadata.contains(EventTimeWatermark.delayKey))
+ assert(eventTimeCols.size === 1)
+ assert(eventTimeCols.head.name === "window")
+ // 2 minutes delay threshold
+ assert(eventTimeCols.head.metadata.getLong(EventTimeWatermark.delayKey) === 120 * 1000)
+ }
+ )
+ }
+ }
+
+ private def buildTestQueryForMultiEventTimeColumns()
+ : (MemoryStream[(String, Long)], MemoryStream[(String, Long)], DataFrame) = {
+ val input1 = MemoryStream[(String, Long)]
+ val input2 = MemoryStream[(String, Long)]
+ val df1 = input1.toDF()
+ .selectExpr("_1 AS id1", "timestamp_seconds(_2) AS ts1")
+ .withWatermark("ts1", "1 minute")
+
+ val df2 = input2.toDF()
+ .selectExpr("_1 AS id2", "timestamp_seconds(_2) AS ts2")
+ .withWatermark("ts2", "2 minutes")
+
+ val joined = df1.join(df2, expr("id1 = id2 AND ts1 = ts2 + INTERVAL 10 SECONDS"), "inner")
+ .selectExpr("id1", "ts1", "ts2")
+ // the output of join contains both ts1 and ts2
+ val dedup = joined.dropDuplicates()
+ .selectExpr("id1", "CAST(ts1 AS LONG) AS ts1", "CAST(ts2 AS LONG) AS ts2")
+
+ (input1, input2, dedup)
+ }
+
+ test("multiple event time columns in an input DataFrame for stateful operator is " +
Review Comment:
Please note that the same query works with compatibility mode as below test case, but there has been no guarantee of correctness. I'd prefer blocking the query for the possible case of correctness issue (hence we do) but I think we may want to also have an escape hatch as well (hence compatibility mode is there).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115352110
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala:
##########
@@ -548,17 +549,131 @@ class EventTimeWatermarkSuite extends StreamTest with BeforeAndAfter with Matche
assert(e.getMessage contains "should not be negative.")
}
- test("the new watermark should override the old one") {
- val df = MemoryStream[(Long, Long)].toDF()
+ private def buildTestQueryForOverridingWatermark(): (MemoryStream[(Long, Long)], DataFrame) = {
+ val input = MemoryStream[(Long, Long)]
+ val df = input.toDF()
.withColumn("first", timestamp_seconds($"_1"))
.withColumn("second", timestamp_seconds($"_2"))
.withWatermark("first", "1 minute")
+ .select("*")
.withWatermark("second", "2 minutes")
+ .groupBy(window($"second", "1 minute"))
+ .count()
- val eventTimeColumns = df.logicalPlan.output
- .filter(_.metadata.contains(EventTimeWatermark.delayKey))
- assert(eventTimeColumns.size === 1)
- assert(eventTimeColumns(0).name === "second")
+ (input, df)
+ }
+
+ test("overriding watermark should not be allowed by default") {
+ val (input, df) = buildTestQueryForOverridingWatermark()
+ testStream(df)(
+ AddData(input, (100L, 200L)),
+ ExpectFailure[AnalysisException](assertFailure = exc => {
+ assert(exc.getMessage.contains("Redefining watermark is disallowed."))
+ assert(exc.getMessage.contains(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key))
+ })
+ )
+ }
+
+ test("overriding watermark should not fail in compatibility mode") {
+ withSQLConf(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key -> "false") {
+ val (input, df) = buildTestQueryForOverridingWatermark()
+ testStream(df)(
+ AddData(input, (100L, 200L)),
+ CheckAnswer(),
+ Execute { query =>
+ val lastExecution = query.lastExecution
+ val aggSaveOperator = lastExecution.executedPlan.collect {
+ case j: StateStoreSaveExec => j
+ }.head
+
+ // - watermark from first definition = 100 - 60 = 40
+ // - watermark from second definition = 200 - 120 = 80
+ // - global watermark = min(40, 60) = 40
+ //
+ // As we see the result, even though we override the watermark definition, the old
+ // definition of watermark still plays to calculate global watermark.
+ //
+ // This is conceptually the right behavior. For operators after the first watermark
+ // definition, the column named "first" is considered as event time column, and for
+ // operators after the second watermark definition, the column named "second" is
+ // considered as event time column. The correct "single" value of watermark satisfying
+ // all operators should be lower bound of both columns "first" and "second".
+ //
+ // That said, this easily leads to incorrect definition - e.g. re-define watermark
+ // against the output of streaming aggregation for append mode. The global watermark
+ // cannot advance. This is the reason we don't allow re-define watermark in new behavior.
+ val expectedWatermarkMs = 40 * 1000
+
+ assert(aggSaveOperator.eventTimeWatermarkForLateEvents === Some(expectedWatermarkMs))
+ assert(aggSaveOperator.eventTimeWatermarkForEviction === Some(expectedWatermarkMs))
+
+ val eventTimeCols = aggSaveOperator.keyExpressions.filter(
+ _.metadata.contains(EventTimeWatermark.delayKey))
+ assert(eventTimeCols.size === 1)
+ assert(eventTimeCols.head.name === "window")
+ // 2 minutes delay threshold
+ assert(eventTimeCols.head.metadata.getLong(EventTimeWatermark.delayKey) === 120 * 1000)
+ }
+ )
+ }
+ }
+
+ private def buildTestQueryForMultiEventTimeColumns()
+ : (MemoryStream[(String, Long)], MemoryStream[(String, Long)], DataFrame) = {
+ val input1 = MemoryStream[(String, Long)]
+ val input2 = MemoryStream[(String, Long)]
+ val df1 = input1.toDF()
+ .selectExpr("_1 AS id1", "timestamp_seconds(_2) AS ts1")
+ .withWatermark("ts1", "1 minute")
+
+ val df2 = input2.toDF()
+ .selectExpr("_1 AS id2", "timestamp_seconds(_2) AS ts2")
+ .withWatermark("ts2", "2 minutes")
+
+ val joined = df1.join(df2, expr("id1 = id2 AND ts1 = ts2 + INTERVAL 10 SECONDS"), "inner")
+ .selectExpr("id1", "ts1", "ts2")
+ // the output of join contains both ts1 and ts2
+ val dedup = joined.dropDuplicates()
+ .selectExpr("id1", "CAST(ts1 AS LONG) AS ts1", "CAST(ts2 AS LONG) AS ts2")
+
+ (input1, input2, dedup)
+ }
+
+ test("multiple event time columns in an input DataFrame for stateful operator is " +
Review Comment:
Please note that the same query runs with compatibility mode as below test case, but there has been no guarantee of correctness. I'd prefer blocking the query for the possible case of correctness issue (hence we do) but I think we may want to also have an escape hatch as well (hence compatibility mode is there).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116635159
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ *
+ * @see [[StateStoreWriter.produceOutputWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ // contains the association for batchId -> (stateful operator ID -> input watermark)
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ /**
+ * Retrieve the available input watermarks for specific node in the plan. Every child will
+ * produce an output watermark, which we capture as input watermark. If the child provides
+ * default watermark value (no watermark info), it is excluded.
+ */
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, originWatermark)
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ inputWatermarks.min
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ DEFAULT_WATERMARK_MS
+ }
+
+ val outputWatermarkMs = node.produceOutputWatermark(finalInputWatermarkMs)
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ val minCurrInputWatermarkMs = inputWatermarks.min
+ minCurrInputWatermarkMs
+ } else {
+ DEFAULT_WATERMARK_MS
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ // In current Spark's logic, event time watermark cannot go down to negative. So even there is
+ // no input watermark for operator, the final input watermark for operator should be 0L.
+ inputWatermarks(batchId).get(stateOpId) match {
+ case Some(wm) => Math.max(0L, wm)
+ case None => throw new IllegalStateException(s"Watermark for batch ID $batchId and " +
+ s"stateOpId $stateOpId is not yet set!")
+ }
+ }
+ }
+
+ override def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId - 1, stateOpId)
+
+ override def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ inputWatermarks.remove(currKey)
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+object WatermarkPropagator {
+ val DEFAULT_WATERMARK_MS = -1L
Review Comment:
(We actually have a limitation that watermark value cannot go down to negative whereas epoch definitely can go down to negative. That's something we eventually need to deal with.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1442092504
> I'm sorry, I'm afraid I still don't get the goal of the test you're proposing. Do you want to see a test case running a query having a bunch of stateful operators without any correctness check, or a test case running a query having reasonable number of stateful operators with correctness check?
More of the latter: a test with lots of stateful operations and it could be simple enough to easily check for correctness. Could you file small ticket and assign it me? I will take a crack it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ninebigbig commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "ninebigbig (via GitHub)" <gi...@apache.org>.
ninebigbig commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1103501006
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -324,6 +344,33 @@ object WatermarkSupport {
}
Some(evictionExpression)
}
+
+ def findEventTimeColumn(
+ attrs: Seq[Attribute],
+ useFirstOccurrence: Boolean): Option[Attribute] = {
+ val eventTimeCols = attrs.filter(_.metadata.contains(EventTimeWatermark.delayKey))
+ if (!useFirstOccurrence) {
+ // There is a case projection leads the same column (same exprId) to appear more than one
+ // time. Allowing them does not hurt the correctness of state row eviction, hence let's start
+ // with allowing them.
+ val eventTimeColsSet = eventTimeCols.map(_.exprId).toSet
+ if (eventTimeColsSet.size > 1) {
+ throw new AnalysisException("More than one event time columns are available. Please " +
+ "ensure there is at most one event time column per stream. event time columns: " +
+ eventTimeCols.mkString("(", ",", ")"))
+ }
+
+ // With above check, even there are multiple columns in eventTimeCols, all columns must be
+ // same.
Review Comment:
typo: must be the same?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1122424081
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
OK. I'll leave a context as a code comment. I agree this is missing and leads some confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1122527231
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
Review Comment:
[c7946f7](https://github.com/apache/spark/pull/39931/commits/c7946f7d6718a9866e97dfcb262243bfb14e0b5a)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109097693
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
Review Comment:
No, that's for compatibility purpose (only when compatibility flag is turned on). PropagateWatermarkSimulator is the default implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119299951
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala:
##########
@@ -90,19 +90,24 @@ trait FlatMapGroupsWithStateExecBase
override def shortName: String = "flatMapGroupsWithState"
- override def shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean = {
+ override def shouldRunAnotherBatch(newInputWatermark: Long): Boolean = {
timeoutConf match {
case ProcessingTimeTimeout =>
true // Always run batches to process timeouts
case EventTimeTimeout =>
// Process another non-data batch only if the watermark has changed in this executed plan
eventTimeWatermarkForEviction.isDefined &&
- newMetadata.batchWatermarkMs > eventTimeWatermarkForEviction.get
+ newInputWatermark > eventTimeWatermarkForEviction.get
case _ =>
false
}
}
+ // There is no guarantee that any of the column in the output is bound to the watermark. The
+ // user function is quite flexible. Hence Spark does not support the stateful operator(s) after
+ // (flat)MapGroupsWithState.
+ override def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = None
Review Comment:
I think this is ok for now, but what is the future plan to support emitting watermarks from flatMapGroupsWithState? It is only an implementation limitation in SS that we cannot support stateful operators after a UDF. I would imagine we can select the first event time column or let the use pick the event time column and use that for watermarking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119697176
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
> The example you provided if the 5 minute window operator outputted a window of 0-5, this window record should be buffered as an entry in 3-6 window range for the 3 minute window.
The representative time of the time window is just to make thing easy to think through timestamp rather than time range (window). It does not mean it is always semantically correct to apply the window time. If you add the 0-5 window into 3-6 window range, it's already producing incorrect output. It's making no sense to add the aggregation for 0-3 into 3-6 window.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1124337029
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,63 +222,140 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
+ private def simulateWatermarkPropagation(plan: SparkPlan): Unit = {
+ val watermarkForPrevBatch = prevOffsetSeqMetadata.map(_.batchWatermarkMs).getOrElse(0L)
+ val watermarkForCurrBatch = offsetSeqMetadata.batchWatermarkMs
+
+ // This is to simulate watermark propagation for late events.
+ watermarkPropagator.propagate(currentBatchId - 1, plan, watermarkForPrevBatch)
+ // This is to simulate watermark propagation for eviction.
+ watermarkPropagator.propagate(currentBatchId, plan, watermarkForCurrBatch)
+ }
+
+ private def inputWatermarkForLateEvents(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForLateEvents(currentBatchId,
+ stateInfo.operatorId))
+ }
+
+ private def inputWatermarkForEviction(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForEviction(currentBatchId, stateInfo.operatorId))
+ }
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+ simulateWatermarkPropagation(plan)
+ plan transform {
+ case s: StateStoreSaveExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: SessionWindowStateStoreSaveExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: SessionWindowStateStoreRestoreExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: StreamingDeduplicateExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case m: FlatMapGroupsWithStateExec if m.stateInfo.isDefined =>
+ m.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(m.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(m.stateInfo.get)
+ )
+
+ case m: FlatMapGroupsInPandasWithStateExec if m.stateInfo.isDefined =>
+ m.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(m.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(m.stateInfo.get)
+ )
+
+ case j: StreamingSymmetricHashJoinExec =>
+ val iwLateEvents = inputWatermarkForLateEvents(j.stateInfo.get)
+ val iwEviction = inputWatermarkForEviction(j.stateInfo.get)
+ j.copy(
+ eventTimeWatermarkForLateEvents = iwLateEvents,
+ eventTimeWatermarkForEviction = iwEviction,
+ stateWatermarkPredicates =
+ StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
+ j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
+ iwEviction, !allowMultipleStatefulOperators)
+ )
+ }
}
}
- override def preparations: Seq[Rule[SparkPlan]] = state +: super.preparations
+ override def preparations: Seq[Rule[SparkPlan]] = Seq(
+ shufflePartitionsRule,
Review Comment:
No pattern seems to be registered for "physical" node. Do you know whether this is somewhat discouraged, or it's just that there was no reason to do so?
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreePatterns.scala
If that's the latter, I'll add some for refactored rules. Otherwise I'll try to consolidate rules. Might be great if we could deal with follow-up ticket/PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1440707038
Looks like we need to rerun the build CI job ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115342363
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
Explained in other comment - https://github.com/apache/spark/pull/39931#issuecomment-1441343786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1441388948
> It's not because the latter operator is dedup. It's because dedup operator gets two event time columns from an input DataFrame, and current late record filtering/eviction logic only handles one event time column per input DataFrame.
@HeartSaVioR isn't the same for 'join + groupBy' too? I.e. `groupBy` stateful operation get two event time columns similar to `dedup()`. What is different here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115384661
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
May be I should read the comment in the test in full. Do you think that will clarify this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115405490
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
I've been thinking through this again... I might be probably overlooked.
My thought was that if there are two event time columns, the state row should wait for the watermark to pass by two values (W > et1 && W > et2) so that further input won't match with the row. But it seems like the condition is OR, not AND. If either W > et1 or W > et2 happens, further input won't match with the row, as W is the lower bound of both columns.
I probably need to revisit this tomorrow again (ideal), or postpone to the next JIRA ticket. If you can help me finding some test/use case which shows expected behavior, that should be awesome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115405490
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
I've been thinking through this again... I might be probably overlooked.
My thought was that if there are two event time columns, the row should wait for the watermark to pass by two values (W > et1 && W > et2) so that further input won't match with the row. But it seems like the condition is OR, not AND. If either W > et1 or W > et2 happens, further input won't match with the row, as W is the lower bound of both columns.
I probably need to revisit this tomorrow again (ideal), or postpone to the next JIRA ticket. If you can help me finding some test/use case which shows expected behavior, that should be awesome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107149596
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -324,6 +344,33 @@ object WatermarkSupport {
}
Some(evictionExpression)
}
+
+ def findEventTimeColumn(
Review Comment:
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107141204
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
Review Comment:
OK. Btw it's specific to this method (not a general comment), right? Since this method is defined as private and we generally don't require code comment for private methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rangadi commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1104859860
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala:
##########
@@ -699,4 +701,18 @@ case class StreamingSymmetricHashJoinExec(
} else {
Nil
}
+
+ // This operator will evict based on the state watermark on both side of inputs; we would like
+ // to let users leverage both sides of event time column for output of join, so the watermark
+ // must be lower bound of both sides of event time column. The lower bound of event time column
+ // for each side is determined by state watermark, hence we take a minimum of (left state
+ // watermark, right state watermark, input watermark) to decide the output watermark.
+ override def produceWatermark(inputWatermarkMs: Long): Long = {
+ val (leftStateWatermark, rightStateWatermark) =
+ StreamingSymmetricHashJoinHelper.getStateWatermark(
+ left.output, right.output, leftKeys, rightKeys, condition.full, Some(inputWatermarkMs))
+
+ (Seq(leftStateWatermark, rightStateWatermark).filter(_.isDefined).map(_.get) ++
Review Comment:
Simpler to write: `(leftStateWatermark ++ leftStateWatermark ++ Some(inputWatermarkMs)).min`
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,52 +219,119 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
+ private def simulateWatermarkPropagation(plan: SparkPlan): Unit = {
+ val watermarkForPrevBatch = prevOffsetSeqMetadata.map(_.batchWatermarkMs).getOrElse(0L)
+ val watermarkForCurrBatch = offsetSeqMetadata.batchWatermarkMs
+
+ // This is to simulate watermark propagation for late events.
+ watermarkPropagator.propagate(currentBatchId - 1, plan, watermarkForPrevBatch)
+ // This is to simulate watermark propagation for eviction.
+ watermarkPropagator.propagate(currentBatchId, plan, watermarkForCurrBatch)
+ }
+
+ private def inputWatermarkForLateEvents(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForLateEvents(currentBatchId,
+ stateInfo.operatorId))
+ }
+
+ private def inputWatermarkForEviction(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForEviction(currentBatchId, stateInfo.operatorId))
+ }
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+ simulateWatermarkPropagation(plan)
+ plan transform {
+ case s: StateStoreSaveExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: SessionWindowStateStoreSaveExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: SessionWindowStateStoreRestoreExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case s: StreamingDeduplicateExec if s.stateInfo.isDefined =>
+ s.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(s.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(s.stateInfo.get)
+ )
+
+ case m: FlatMapGroupsWithStateExec if m.stateInfo.isDefined =>
+ m.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(m.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(m.stateInfo.get)
+ )
+
+ case m: FlatMapGroupsInPandasWithStateExec if m.stateInfo.isDefined =>
+ m.copy(
+ eventTimeWatermarkForLateEvents = inputWatermarkForLateEvents(m.stateInfo.get),
+ eventTimeWatermarkForEviction = inputWatermarkForEviction(m.stateInfo.get)
+ )
+
+ case j: StreamingSymmetricHashJoinExec =>
+ val iwLateEvents = inputWatermarkForLateEvents(j.stateInfo.get)
+ val iwEviction = inputWatermarkForEviction(j.stateInfo.get)
+ j.copy(
+ eventTimeWatermarkForLateEvents = iwLateEvents,
+ eventTimeWatermarkForEviction = iwEviction,
+ stateWatermarkPredicates =
+ StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
+ j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
+ iwEviction)
+ )
+ }
}
}
- override def preparations: Seq[Rule[SparkPlan]] = state +: super.preparations
+ override def preparations: Seq[Rule[SparkPlan]] = Seq(
+ shufflePartitionsRule, convertLocalLimitRule, stateOpIdRule, watermarkPropagationRule
Review Comment:
Minor: Could you move these to one per line? Since the order matters, it is more readable to see the order.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
Review Comment:
In fact this will be called for every node in the plan, right (stateful or not)?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala:
##########
@@ -216,52 +219,119 @@ class IncrementalExecution(
keys,
child,
Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None)
case m: FlatMapGroupsWithStateExec =>
// We set this to true only for the first batch of the streaming query.
val hasInitialState = (currentBatchId == 0L && m.hasInitialState)
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None,
hasInitialState = hasInitialState
)
case m: FlatMapGroupsInPandasWithStateExec =>
m.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction)
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
)
case j: StreamingSymmetricHashJoinExec =>
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
- eventTimeWatermarkForLateEvents = Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
- stateWatermarkPredicates =
- StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
- j.left.output, j.right.output, j.leftKeys, j.rightKeys, j.condition.full,
- Some(eventTimeWatermarkForEviction)))
+ eventTimeWatermarkForLateEvents = None,
+ eventTimeWatermarkForEviction = None
+ )
case l: StreamingGlobalLimitExec =>
l.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
outputMode = Some(outputMode))
+ }
+ }
- case StreamingLocalLimitExec(limit, child) if hasNoStatefulOp(child) =>
- // Optimize limit execution by replacing StreamingLocalLimitExec (consumes the iterator
- // completely) to LocalLimitExec (does not consume the iterator) when the child plan has
- // no stateful operator (i.e., consuming the iterator is not needed).
- LocalLimitExec(limit, child)
+ val watermarkPropagationRule = new Rule[SparkPlan] {
+ private def simulateWatermarkPropagation(plan: SparkPlan): Unit = {
+ val watermarkForPrevBatch = prevOffsetSeqMetadata.map(_.batchWatermarkMs).getOrElse(0L)
+ val watermarkForCurrBatch = offsetSeqMetadata.batchWatermarkMs
+
+ // This is to simulate watermark propagation for late events.
+ watermarkPropagator.propagate(currentBatchId - 1, plan, watermarkForPrevBatch)
+ // This is to simulate watermark propagation for eviction.
+ watermarkPropagator.propagate(currentBatchId, plan, watermarkForCurrBatch)
+ }
+
+ private def inputWatermarkForLateEvents(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForLateEvents(currentBatchId,
+ stateInfo.operatorId))
+ }
+
+ private def inputWatermarkForEviction(stateInfo: StatefulOperatorStateInfo): Option[Long] = {
+ Some(watermarkPropagator.getInputWatermarkForEviction(currentBatchId, stateInfo.operatorId))
+ }
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+ simulateWatermarkPropagation(plan)
Review Comment:
How about moving this into case statements below? Or move it into `inputWatermarkForLateEvents()` and `inputWatermarkForEviction()`? That way it will only be called for stateful operators.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, originWatermark)
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ inputWatermarks.min
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ DEFAULT_WATERMARK_MS
+ }
+
+ val outputWatermarkMs = node.produceWatermark(finalInputWatermarkMs)
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ val minCurrInputWatermarkMs = inputWatermarks.min
+ minCurrInputWatermarkMs
+ } else {
+ DEFAULT_WATERMARK_MS
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
Review Comment:
Clarification: How does this ensure this is propagated through the entire plan? Does this plan arg here matter?
- `populate()` is called for every node
- `doSimulate()` is called only once per batch.
- Assuming `transformUp()` above only traverses partial tree, how do ensure we traverse the entire tree the first time?
- Is the sync node the root and and thus we traverse the entire tree in the first iteration?
Could you attach the debug log at line 220 and 221 from a unit test?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
Review Comment:
What does this contain? Add a comment.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -324,6 +344,33 @@ object WatermarkSupport {
}
Some(evictionExpression)
}
+
+ def findEventTimeColumn(
Review Comment:
Add method documentation since this is used outside.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala:
##########
@@ -699,4 +701,18 @@ case class StreamingSymmetricHashJoinExec(
} else {
Nil
}
+
+ // This operator will evict based on the state watermark on both side of inputs; we would like
+ // to let users leverage both sides of event time column for output of join, so the watermark
+ // must be lower bound of both sides of event time column. The lower bound of event time column
+ // for each side is determined by state watermark, hence we take a minimum of (left state
+ // watermark, right state watermark, input watermark) to decide the output watermark.
+ override def produceWatermark(inputWatermarkMs: Long): Long = {
Review Comment:
For better context: What node does this `inputWatermarkMs` correspond to?
(will check `StreamingSymmetricHashJoinHelper.getStateWatermark()` as well).
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
Review Comment:
Better to make it an `object` (it makes it clear).
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
Review Comment:
Add a comment on what this does. i.e. what does it return?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
Review Comment:
Btw, in physical plan, are children executed before the parent? (E.g. in a stream-stream join, are left & right children of join node?)
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,20 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ */
+ def produceWatermark(inputWatermarkMs: Long): Long
Review Comment:
How about renaming it to `produceOutputWatermark()` for clarity?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, originWatermark)
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ inputWatermarks.min
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ DEFAULT_WATERMARK_MS
+ }
+
+ val outputWatermarkMs = node.produceWatermark(finalInputWatermarkMs)
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ val minCurrInputWatermarkMs = inputWatermarks.min
+ minCurrInputWatermarkMs
+ } else {
+ DEFAULT_WATERMARK_MS
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ // In current Spark's logic, event time watermark cannot go down to negative. So even there is
+ // no input watermark for operator, the final input watermark for operator should be 0L.
+ val opWatermark = inputWatermarks(batchId).get(stateOpId)
Review Comment:
Style: use match for better readability (here and other places):
```scala
inputWatermarks(batchId).get(stateOpId) match {
case Some(wm) => Math.max(0, wm)
case None => throw IllegalStateException(...)
}
```
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -809,6 +860,10 @@ case class SessionWindowStateStoreSaveExec(
newNumRowsUpdated = stateOpProgress.numRowsUpdated,
newNumRowsDroppedByWatermark = numRowsDroppedByWatermark)
}
+
+ // This operator will evict based on min input watermark and ensure it will be minimum of
+ // the event time value for the output so far (including output from eviction).
+ override def produceWatermark(inputWatermarkMs: Long): Long = inputWatermarkMs
Review Comment:
How about making this the default implementation so that no all stateful operators need to define this default behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107145581
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, originWatermark)
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ inputWatermarks.min
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ DEFAULT_WATERMARK_MS
+ }
+
+ val outputWatermarkMs = node.produceWatermark(finalInputWatermarkMs)
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ val minCurrInputWatermarkMs = inputWatermarks.min
+ minCurrInputWatermarkMs
+ } else {
+ DEFAULT_WATERMARK_MS
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+ doSimulate(batchId, plan, originWatermark)
Review Comment:
populate() is called once for entire physical plan. Then others will be clarified automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1106889221
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
Review Comment:
Yeah I don't see the path this method is called concurrently. (The built-in implementations also don't deal with concurrent map.) While I think this interface and implementation will be private classes, I agree that making thread-safety be explicit would help callers. Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zsxwing commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "zsxwing (via GitHub)" <gi...@apache.org>.
zsxwing commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1127324257
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceOutputWatermark(input watermark)`.
+ *
+ * @see [[StateStoreWriter.produceOutputWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ *
+ * We skip simulation of propagation for the value of watermark as 0. Input watermark for every
+ * operator will be 0. (This may not be expected for the case op.produceOutputWatermark returns
+ * higher than the input watermark, but it won't happen in most practical cases.)
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ // We use treemap to sort the key (batchID) and evict old batch IDs efficiently.
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ // contains the association for batchId -> (stateful operator ID -> input watermark)
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Option[Long]]] =
+ mutable.Map[Long, Map[Long, Option[Long]]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ /**
+ * Retrieve the available input watermarks for specific node in the plan. Every child will
+ * produce an output watermark, which we capture as input watermark. If the child provides
+ * default watermark value (no watermark info), it is excluded.
+ */
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Option[Long]]): Seq[Long] = {
+ node.children.flatMap { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ // Since we use flatMap here, this will exclude children from watermark calculation
+ // which don't have watermark information.
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Option[Long]]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Option[Long]]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
+ case node: EventTimeWatermarkExec =>
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ if (inputWatermarks.nonEmpty) {
+ throw new AnalysisException("Redefining watermark is disallowed. You can set the " +
+ s"config '${SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key}' to 'false' to restore " +
+ "the previous behavior. Note that multiple stateful operators will be disallowed.")
+ }
+
+ nodeToOutputWatermark.put(node.id, Some(originWatermark))
+ node
+
+ case node: StateStoreWriter =>
+ val stOpId = node.stateInfo.get.operatorId
+ statefulOperatorIdToNodeId.put(stOpId, node.id)
+
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ Some(inputWatermarks.min)
+ } else {
+ // We can't throw exception here, as we allow stateful operator to process without
+ // watermark. E.g. streaming aggregation with update/complete mode.
+ None
+ }
+
+ val outputWatermarkMs = finalInputWatermarkMs.flatMap { wm =>
+ node.produceOutputWatermark(wm)
+ }
+ nodeToOutputWatermark.put(node.id, outputWatermarkMs)
+ nextStatefulOperatorToWatermark.put(stOpId, finalInputWatermarkMs)
+ node
+
+ case node =>
+ // pass-through, but also consider multiple children like the case of union
+ val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
+ val finalInputWatermarkMs = if (inputWatermarks.nonEmpty) {
+ Some(inputWatermarks.min)
+ } else {
+ None
+ }
+
+ nodeToOutputWatermark.put(node.id, finalInputWatermarkMs)
+ node
+ }
+
+ inputWatermarks.put(batchId, nextStatefulOperatorToWatermark.toMap)
+ batchIdToWatermark.put(batchId, originWatermark)
+
+ logDebug(s"global watermark for batch ID $batchId is set to $originWatermark")
+ logDebug(s"input watermarks for batch ID $batchId is set to $nextStatefulOperatorToWatermark")
+ }
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ logDebug(s"watermark for batch ID $batchId is received as $originWatermark, " +
+ s"call site: ${Utils.getCallSite().longForm}")
+
+ if (originWatermark == 0) {
+ logDebug(s"skipping the propagation for batch $batchId as origin watermark is 0.")
+ batchIdToWatermark.put(batchId, 0L)
+ inputWatermarks.put(batchId, Map.empty[Long, Option[Long]])
+ } else {
+ doSimulate(batchId, plan, originWatermark)
+ }
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ inputWatermarks(batchId).get(stateOpId) match {
+ case Some(wmOpt) =>
+ // In current Spark's logic, event time watermark cannot go down to negative. So even
+ // there is no input watermark for operator, the final input watermark for operator should
+ // be 0L.
+ Math.max(wmOpt.getOrElse(0L), 0L)
+ case None =>
+ if (batchIdToWatermark.get(batchId) == 0L) {
+ // We skip the propagation when the origin watermark is produced as 0L. This is safe,
+ // as output watermark is not expected to be later than the input watermark. That said,
+ // all operators would have the input watermark as 0L.
+ 0L
+ } else {
+ throw new IllegalStateException(s"Watermark for batch ID $batchId and " +
+ s"stateOpId $stateOpId is not yet set!")
+ }
+ }
+ }
+ }
+
+ override def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = {
+ // We use watermark for previous microbatch to determine late events.
+ getInputWatermark(batchId - 1, stateOpId)
+ }
+
+ override def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ inputWatermarks.remove(currKey)
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+object WatermarkPropagator {
+ def apply(conf: SQLConf): WatermarkPropagator = {
+ if (conf.getConf(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE)) {
Review Comment:
I just realized `STATEFUL_OPERATOR_ALLOW_MULTIPLE` would be released in 3.4.0. Can we create a separate internal flag for this new change that's going to be released in 3.5.0? I understand that adding a flag is ugly but it would be great to use different flags to track different changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109092081
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/**
+ * Interface for propagating watermark. The implementation is not required to be thread-safe,
+ * as all methods are expected to be called from the query execution thread.
+ * (The guarantee may change on further improvements on Structured Streaming - update
+ * implementations if we change the guarantee.)
+ */
+sealed trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+object NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
Review Comment:
Maybe add a small comment to explain why TreeMap is better here for purge since we can iterate keys in sorted order ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1109076615
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Long = inputWatermarkMs
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115405490
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala:
##########
@@ -463,6 +437,442 @@ class MultiStatefulOperatorsSuite
)
}
+ test("stream-stream time interval left outer join -> aggregation, append mode") {
+ val input1 = MemoryStream[(String, Timestamp)]
+ val input2 = MemoryStream[(String, Timestamp)]
+
+ val s1 = input1.toDF()
+ .selectExpr("_1 AS id1", "_2 AS timestamp1")
+ .withWatermark("timestamp1", "0 seconds")
+ .as("s1")
+
+ val s2 = input2.toDF()
+ .selectExpr("_1 AS id2", "_2 AS timestamp2")
+ .withWatermark("timestamp2", "0 seconds")
+ .as("s2")
+
+ val s3 = s1.join(s2, expr("s1.id1 = s2.id2 AND (s1.timestamp1 BETWEEN " +
+ "s2.timestamp2 - INTERVAL 1 hour AND s2.timestamp2 + INTERVAL 1 hour)"), "leftOuter")
+
+ val agg = s3.groupBy(window($"timestamp1", "10 minutes"))
Review Comment:
I've been thinking through this again... I might be probably overlooked.
My thought was that if there are two event time columns, the state row should wait for the watermark to pass by two values (W > et1 && W > et2) so that further input won't match with the row. But it seems like the condition is OR, not AND. If either W > et1 or W > et2 happens, further input won't match with the row, as W is the lower bound of both columns.
I probably need to revisit this tomorrow again (ideal), or postpone to the next JIRA ticket. If you can help me finding some test/use case which shows expected behavior (and probably an edge-case), that should be awesome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1115344419
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala:
##########
@@ -548,17 +549,131 @@ class EventTimeWatermarkSuite extends StreamTest with BeforeAndAfter with Matche
assert(e.getMessage contains "should not be negative.")
}
- test("the new watermark should override the old one") {
- val df = MemoryStream[(Long, Long)].toDF()
+ private def buildTestQueryForOverridingWatermark(): (MemoryStream[(Long, Long)], DataFrame) = {
+ val input = MemoryStream[(Long, Long)]
+ val df = input.toDF()
.withColumn("first", timestamp_seconds($"_1"))
.withColumn("second", timestamp_seconds($"_2"))
.withWatermark("first", "1 minute")
+ .select("*")
.withWatermark("second", "2 minutes")
+ .groupBy(window($"second", "1 minute"))
+ .count()
- val eventTimeColumns = df.logicalPlan.output
- .filter(_.metadata.contains(EventTimeWatermark.delayKey))
- assert(eventTimeColumns.size === 1)
- assert(eventTimeColumns(0).name === "second")
+ (input, df)
+ }
+
+ test("overriding watermark should not be allowed by default") {
+ val (input, df) = buildTestQueryForOverridingWatermark()
+ testStream(df)(
+ AddData(input, (100L, 200L)),
+ ExpectFailure[AnalysisException](assertFailure = exc => {
+ assert(exc.getMessage.contains("Redefining watermark is disallowed."))
+ assert(exc.getMessage.contains(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key))
+ })
+ )
+ }
+
+ test("overriding watermark should not fail in compatibility mode") {
+ withSQLConf(SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key -> "false") {
+ val (input, df) = buildTestQueryForOverridingWatermark()
+ testStream(df)(
+ AddData(input, (100L, 200L)),
+ CheckAnswer(),
+ Execute { query =>
+ val lastExecution = query.lastExecution
+ val aggSaveOperator = lastExecution.executedPlan.collect {
+ case j: StateStoreSaveExec => j
+ }.head
+
+ // - watermark from first definition = 100 - 60 = 40
+ // - watermark from second definition = 200 - 120 = 80
+ // - global watermark = min(40, 60) = 40
+ //
+ // As we see the result, even though we override the watermark definition, the old
+ // definition of watermark still plays to calculate global watermark.
+ //
+ // This is conceptually the right behavior. For operators after the first watermark
+ // definition, the column named "first" is considered as event time column, and for
+ // operators after the second watermark definition, the column named "second" is
+ // considered as event time column. The correct "single" value of watermark satisfying
+ // all operators should be lower bound of both columns "first" and "second".
+ //
+ // That said, this easily leads to incorrect definition - e.g. re-define watermark
+ // against the output of streaming aggregation for append mode. The global watermark
+ // cannot advance. This is the reason we don't allow re-define watermark in new behavior.
+ val expectedWatermarkMs = 40 * 1000
+
+ assert(aggSaveOperator.eventTimeWatermarkForLateEvents === Some(expectedWatermarkMs))
+ assert(aggSaveOperator.eventTimeWatermarkForEviction === Some(expectedWatermarkMs))
+
+ val eventTimeCols = aggSaveOperator.keyExpressions.filter(
+ _.metadata.contains(EventTimeWatermark.delayKey))
+ assert(eventTimeCols.size === 1)
+ assert(eventTimeCols.head.name === "window")
+ // 2 minutes delay threshold
+ assert(eventTimeCols.head.metadata.getLong(EventTimeWatermark.delayKey) === 120 * 1000)
+ }
+ )
+ }
+ }
+
+ private def buildTestQueryForMultiEventTimeColumns()
+ : (MemoryStream[(String, Long)], MemoryStream[(String, Long)], DataFrame) = {
+ val input1 = MemoryStream[(String, Long)]
+ val input2 = MemoryStream[(String, Long)]
+ val df1 = input1.toDF()
+ .selectExpr("_1 AS id1", "timestamp_seconds(_2) AS ts1")
+ .withWatermark("ts1", "1 minute")
+
+ val df2 = input2.toDF()
+ .selectExpr("_1 AS id2", "timestamp_seconds(_2) AS ts2")
+ .withWatermark("ts2", "2 minutes")
+
+ val joined = df1.join(df2, expr("id1 = id2 AND ts1 = ts2 + INTERVAL 10 SECONDS"), "inner")
+ .selectExpr("id1", "ts1", "ts2")
+ // the output of join contains both ts1 and ts2
+ val dedup = joined.dropDuplicates()
+ .selectExpr("id1", "CAST(ts1 AS LONG) AS ts1", "CAST(ts2 AS LONG) AS ts2")
+
+ (input1, input2, dedup)
+ }
+
+ test("multiple event time columns in an input DataFrame for stateful operator is " +
Review Comment:
One sketched idea would be... we could create expressions for all event time columns and account them for late record filtering / eviction. If we'd like to see this work we could file a JIRA ticket and check for prioritization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1441390163
For the test case of join + groupBy, we picked one event time column for time window and discarded other one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1106896139
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala:
##########
@@ -699,4 +701,18 @@ case class StreamingSymmetricHashJoinExec(
} else {
Nil
}
+
+ // This operator will evict based on the state watermark on both side of inputs; we would like
+ // to let users leverage both sides of event time column for output of join, so the watermark
+ // must be lower bound of both sides of event time column. The lower bound of event time column
+ // for each side is determined by state watermark, hence we take a minimum of (left state
+ // watermark, right state watermark, input watermark) to decide the output watermark.
+ override def produceWatermark(inputWatermarkMs: Long): Long = {
+ val (leftStateWatermark, rightStateWatermark) =
+ StreamingSymmetricHashJoinHelper.getStateWatermark(
+ left.output, right.output, leftKeys, rightKeys, condition.full, Some(inputWatermarkMs))
+
+ (Seq(leftStateWatermark, rightStateWatermark).filter(_.isDefined).map(_.get) ++
Review Comment:
Thanks, will try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107127363
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala:
##########
@@ -699,4 +701,18 @@ case class StreamingSymmetricHashJoinExec(
} else {
Nil
}
+
+ // This operator will evict based on the state watermark on both side of inputs; we would like
+ // to let users leverage both sides of event time column for output of join, so the watermark
+ // must be lower bound of both sides of event time column. The lower bound of event time column
+ // for each side is determined by state watermark, hence we take a minimum of (left state
+ // watermark, right state watermark, input watermark) to decide the output watermark.
+ override def produceWatermark(inputWatermarkMs: Long): Long = {
Review Comment:
That's the minimum of the output watermark (input watermark for this operator) in upstream operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1107142832
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkPropagator.scala:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming
+
+import java.{util => jutil}
+
+import scala.collection.mutable
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.streaming.WatermarkPropagator.DEFAULT_WATERMARK_MS
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
+
+/** Interface for propagating watermark. */
+trait WatermarkPropagator {
+ /**
+ * Request to propagate watermark among operators based on origin watermark value. The result
+ * should be input watermark per stateful operator, which Spark will request the value by calling
+ * getInputWatermarkXXX with operator ID.
+ *
+ * It is recommended for implementation to cache the result, as Spark can request the propagation
+ * multiple times with the same batch ID and origin watermark value.
+ */
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
+
+ /** Provide the calculated input watermark for late events for given stateful operator. */
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
+
+ /** Provide the calculated input watermark for eviction for given stateful operator. */
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
+
+ /**
+ * Request to clean up cached result on propagation. Spark will call this method when the given
+ * batch ID will be likely to be not re-executed.
+ */
+ def purge(batchId: Long): Unit
+}
+
+/**
+ * Do nothing. This is dummy implementation to help creating a dummy IncrementalExecution instance.
+ */
+class NoOpWatermarkPropagator extends WatermarkPropagator {
+ def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {}
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long = Long.MinValue
+ def purge(batchId: Long): Unit = {}
+}
+
+/**
+ * This implementation uses a single global watermark for late events and eviction.
+ *
+ * This implementation provides the behavior before Structured Streaming supports multiple stateful
+ * operators. (prior to SPARK-40925) This is only used for compatibility mode.
+ */
+class UseSingleWatermarkPropagator extends WatermarkPropagator {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ override def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ if (batchId < 0) {
+ // no-op
+ } else if (isInitialized(batchId)) {
+ val cached = batchIdToWatermark.get(batchId)
+ assert(cached == originWatermark,
+ s"Watermark has been changed for the same batch ID! Batch ID: $batchId, " +
+ s"Value in cache: $cached, value given: $originWatermark")
+ } else {
+ batchIdToWatermark.put(batchId, originWatermark)
+ }
+ }
+
+ private def getInputWatermark(batchId: Long, stateOpId: Long): Long = {
+ if (batchId < 0) {
+ 0
+ } else {
+ assert(isInitialized(batchId), s"Watermark for batch ID $batchId is not yet set!")
+ batchIdToWatermark.get(batchId)
+ }
+ }
+
+ def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long =
+ getInputWatermark(batchId, stateOpId)
+
+ override def purge(batchId: Long): Unit = {
+ val keyIter = batchIdToWatermark.keySet().iterator()
+ var stopIter = false
+ while (keyIter.hasNext && !stopIter) {
+ val currKey = keyIter.next()
+ if (currKey <= batchId) {
+ keyIter.remove()
+ } else {
+ stopIter = true
+ }
+ }
+ }
+}
+
+/**
+ * This implementation simulates propagation of watermark among operators.
+ *
+ * The simulation algorithm traverses the physical plan tree via post-order (children first) to
+ * calculate (input watermark, output watermark) for all nodes.
+ *
+ * For each node, below logic is applied:
+ *
+ * - Input watermark for specific node is decided by `min(input watermarks from all children)`.
+ * -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded.
+ * -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS.
+ * - Output watermark for specific node is decided as following:
+ * -- watermark nodes: origin watermark value
+ * This could be individual origin watermark value, but we decide to retain global watermark
+ * to keep the watermark model be simple.
+ * -- stateless nodes: same as input watermark
+ * -- stateful nodes: the return value of `op.produceWatermark(input watermark)`.
+ * @see [[StateStoreWriter.produceWatermark]]
+ *
+ * Note that this implementation will throw an exception if watermark node sees a valid input
+ * watermark from children, meaning that we do not support re-definition of watermark.
+ *
+ * Once the algorithm traverses the physical plan tree, the association between stateful operator
+ * and input watermark will be constructed. Spark will request the input watermark for specific
+ * stateful operator, which this implementation will give the value from the association.
+ */
+class PropagateWatermarkSimulator extends WatermarkPropagator with Logging {
+ private val batchIdToWatermark: jutil.TreeMap[Long, Long] = new jutil.TreeMap[Long, Long]()
+ private val inputWatermarks: mutable.Map[Long, Map[Long, Long]] =
+ mutable.Map[Long, Map[Long, Long]]()
+
+ private def isInitialized(batchId: Long): Boolean = batchIdToWatermark.containsKey(batchId)
+
+ private def getInputWatermarks(
+ node: SparkPlan,
+ nodeToOutputWatermark: mutable.Map[Int, Long]): Seq[Long] = {
+ node.children.map { child =>
+ nodeToOutputWatermark.getOrElse(child.id, {
+ throw new IllegalStateException(
+ s"watermark for the node ${child.id} should be registered")
+ })
+ }.filter { case curr =>
+ // This path is to exclude children from watermark calculation
+ // which don't have watermark information
+ curr != DEFAULT_WATERMARK_MS
+ }
+ }
+
+ private def doSimulate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit = {
+ val statefulOperatorIdToNodeId = mutable.HashMap[Long, Int]()
+ val nodeToOutputWatermark = mutable.HashMap[Int, Long]()
+ val nextStatefulOperatorToWatermark = mutable.HashMap[Long, Long]()
+
+ // This calculation relies on post-order traversal of the query plan.
+ plan.transformUp {
Review Comment:
Yes, the physical plan is a tree which leaves are sources (or equivalent one) and root is mostly sink (or final operator).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119692932
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
While we say "representative time" of time window, we still need to reason about the semantic for chained time window aggregation, based on how time windows are "inclusive" in bigger time window.
For example, if we suppose to have a query having 5 mins time window -> 3 mins time window. Are we really sure having any output in 3 mins time window is valid? Where the output of 5 mins time window have to bound for next 3 mins time window? Why we need to worry about the case which just doesn't make sense at all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1120512476
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
@HeartSaVioR thanks for the detailed explanation. Can you add we add this information as a comment in the code so we don't forget the rationale of why we are doing this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119407026
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
Your example is not consistent. If you want to advance watermark "after" you produce output, both 5 minutes and 10 minutes window operator watermark should be 0.
Even we provide the output watermark as same as input watermark, it does not violate the definition of watermark, lower bound of the "output". Advancing watermark as 3 for 5 minutes window does not break the logic. Further output will be at least 5, not less than 3, hence the previous advancement of watermark is never violated. Current watermark logic has been working like this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jerrypeng commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "jerrypeng (via GitHub)" <gi...@apache.org>.
jerrypeng commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1116567932
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Long = inputWatermarkMs
Review Comment:
Can this output an optional for operators that don't output a watermark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1119402665
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -324,6 +349,41 @@ object WatermarkSupport {
}
Some(evictionExpression)
}
+
+ /**
+ * Find the column which is marked as "event time" column.
+ *
+ * If there are multiple event time columns in given column list, the behavior depends on the
+ * parameter `useFirstOccurrence`. If it's set to true, the first occurred column will be
+ * returned. If not, this method will throw an AnalysisException as it is not allowed to have
+ * multiple event time columns.
+ */
+ def findEventTimeColumn(
+ attrs: Seq[Attribute],
+ useFirstOccurrence: Boolean): Option[Attribute] = {
Review Comment:
Good suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #39931:
URL: https://github.com/apache/spark/pull/39931#issuecomment-1441343786
> One thing that is not clear is how 'join + dedup()' is not allwowd, but 'join + groupBy,count' is allowed.
It's not because the latter operator is dedup. It's because dedup operator gets two event time columns from an input DataFrame, and current late record filtering/eviction logic only handles one event time column per input DataFrame. It has been picking the first occurrence (simply assuming there is only one event time column which is actually incorrect), but we should have considered all event time columns. e.g. et1 00:10 et2 00:30 W=00:20 -> the row should not be evicted as further row can be matched with this row (et2 is still earlier than W), but we evict it.
We may be able to address this in (near) future, but I'd like to let it be out of scope for this effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1122527231
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
Review Comment:
fe65ef0
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
fe65ef0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #39931: [SPARK-42376][SS] Introduce watermark propagation among operators
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #39931:
URL: https://github.com/apache/spark/pull/39931#discussion_r1122527304
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala:
##########
@@ -96,6 +98,25 @@ trait StateStoreReader extends StatefulOperator {
/** An operator that writes to a StateStore. */
trait StateStoreWriter extends StatefulOperator with PythonSQLMetrics { self: SparkPlan =>
+ /**
+ * Produce the output watermark for given input watermark (ms).
+ *
+ * In most cases, this is same as the criteria of state eviction, as most stateful operators
+ * produce the output from two different kinds:
+ *
+ * 1. without buffering
+ * 2. with buffering (state)
+ *
+ * The state eviction happens when event time exceeds a "certain threshold of timestamp", which
+ * denotes a lower bound of event time values for output (output watermark).
+ *
+ * The default implementation provides the input watermark as it is. Most built-in operators
+ * will evict based on min input watermark and ensure it will be minimum of the event time value
+ * for the output so far (including output from eviction). Operators which behave differently
+ * (e.g. different criteria on eviction) must override this method.
+ */
+ def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = Some(inputWatermarkMs)
Review Comment:
[c7946f7](https://github.com/apache/spark/pull/39931/commits/c7946f7d6718a9866e97dfcb262243bfb14e0b5a)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org