You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2019/06/17 07:04:00 UTC

[jira] [Created] (SPARK-28074) [SS] Document caveats on using multiple stateful operations in single query

Jungtaek Lim created SPARK-28074:
------------------------------------

             Summary: [SS] Document caveats on using multiple stateful operations in single query
                 Key: SPARK-28074
                 URL: https://issues.apache.org/jira/browse/SPARK-28074
             Project: Spark
          Issue Type: Documentation
          Components: Structured Streaming
    Affects Versions: 3.0.0
            Reporter: Jungtaek Lim


Please refer [https://lists.apache.org/thread.html/cc6489a19316e7382661d305fabd8c21915e5faf6a928b4869ac2b4a@%3Cdev.spark.apache.org%3E] to see rationalization of the issue.

tl;dr. Using multiple stateful operators without fully understanding how global watermark affects the query brings correctness issue.

While we may not want to fully block the operations given some users could leverage it with understanding of global watermark, it would be necessary for end users to warn on such breaking-correctness possibility, and provide alternative(s).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org