[jira] [Created] (SPARK-47698) Current doc section handling-late-data-and-watermarking is misleading after the support for multiple stateful operators

["Lingeshwaran Radhakrishnan (Jira)" <ji...@apache.org>]

Lingeshwaran Radhakrishnan created SPARK-47698:
--------------------------------------------------

             Summary: Current doc section handling-late-data-and-watermarking is misleading after the support for multiple stateful operators
                 Key: SPARK-47698
                 URL: https://issues.apache.org/jira/browse/SPARK-47698
             Project: Spark
          Issue Type: Documentation
          Components: Documentation
    Affects Versions: 3.4.0
            Reporter: Lingeshwaran Radhakrishnan
         Attachments: image-2024-04-02-15-04-34-287.png

[This section|https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#handling-late-data-and-watermarking] of the doc which explains the nuances of handling late records using beautiful diagrams has gone out of context and a bit misleading after *multiple stateful operators'* support was introduced with https://issues.apache.org/jira/browse/SPARK-40925 

!image-2024-04-02-15-01-25-523.png!

 

Previously watermark is applied for batch N to filter out inputs in batch N. With support for multiple stateful operators, the watermark is applied for batch N-1 instead. The doc section above should reflect this new behavior to avoid confusion.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org