[GitHub] [spark] gaborgsomogyi commented on a change in pull request #24890: [SPARK-28074][DOC][SS] Document caveats on using multiple stateful operations in single query

[GitBox <gi...@apache.org>]

gaborgsomogyi commented on a change in pull request #24890: [SPARK-28074][DOC][SS] Document caveats on using multiple stateful operations in single query
URL: https://github.com/apache/spark/pull/24890#discussion_r298525235
 
 

 ##########
 File path: docs/structured-streaming-programming-guide.md
 ##########
 @@ -3146,6 +3146,17 @@ See [Input Sources](#input-sources) and [Output Sinks](#output-sinks) sections f
       - After `coalesce`, the number of (reduced) tasks will be kept unless another shuffle happens.
   - `spark.sql.streaming.stateStore.providerClass`: To read the previous state of the query properly, the class of state store provider should be unchanged.
   - `spark.sql.streaming.multipleWatermarkPolicy`: Modification of this would lead inconsistent watermark value when query contains multiple watermarks, hence the policy should be unchanged.
+- Structured Streaming uses `global watermark` which might impact on query having multiple stateful operations.
+  - Stateful operators: aggregation, deduplication, stream-stream join, (flat)mapGroupsWithState
+  - You should be able to answer below questions for your query to get correct outputs:
 
 Review comment:
   Not sure this will be understood by users (I know the details since played a lot with it but also a bit struggling :)). I would focus on global watermark and maybe the simplest discard example + mentioning this may apply other situations.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org