[GitHub] [spark] HeartSaVioR commented on a change in pull request #24890: [SPARK-28074][DOC][SS] Document caveats on using multiple stateful operations in single query

[GitBox <gi...@apache.org>]

HeartSaVioR commented on a change in pull request #24890: [SPARK-28074][DOC][SS] Document caveats on using multiple stateful operations in single query
URL: https://github.com/apache/spark/pull/24890#discussion_r295060594
 
 

 ##########
 File path: docs/structured-streaming-programming-guide.md
 ##########
 @@ -3146,6 +3146,17 @@ See [Input Sources](#input-sources) and [Output Sinks](#output-sinks) sections f
       - After `coalesce`, the number of (reduced) tasks will be kept unless another shuffle happens.
   - `spark.sql.streaming.stateStore.providerClass`: To read the previous state of the query properly, the class of state store provider should be unchanged.
   - `spark.sql.streaming.multipleWatermarkPolicy`: Modification of this would lead inconsistent watermark value when query contains multiple watermarks, hence the policy should be unchanged.
+- Structured Streaming uses `global watermark` which might impact on query having multiple stateful operations.
+  - Stateful operators: aggregation, deduplication, stream-stream join, (flat)mapGroupsWithState
+  - You should be able to answer below questions for your query to get correct outputs:
+    - How global watermark is calculated on your query?
+    - How global watermark is applied to each stateful operator?
+    - Is there any intermediate output being discarded as "late input" due to watermark?
+  - Fail to answer above questions might lead to incorrect outputs - e.g. intermediate outputs being discarded. 
+  - One of "alternative" approach is breaking down your query into multiple chained queries, each per stateful operation.
+    - Each query must guarantee "end-to-end" exactly once, otherwise intermediate outputs can be duplicated which leads to incorrect outputs.
+- Only 'Append mode' can be "semantically" correct for a query having multiple stateful operations.
+  - In 'Update mode', downstream stateful operator cannot distinguish whether the input is new, or updated.
 
 Review comment:
   Honestly I agree with @srowen regarding screenshot, as it doesn't draw some UI like table. They're just bullet points with proper indentation, and if we add it to PR description I should update the screenshot whenever I reflect review comments, which sounds redundant.
   
   Regarding too many bullet-pointed sentences, I agree there're some unnecessary bullet points. I'll concat. Any feedbacks on nuance, writing style, etc are welcome, as I'm not native speaker of English.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org