[jira] [Created] (SPARK-43183) Move update event on idleness in streaming query listener to separate callback method

["Jungtaek Lim (Jira)" <ji...@apache.org>]

Jungtaek Lim created SPARK-43183:
------------------------------------

             Summary: Move update event on idleness in streaming query listener to separate callback method
                 Key: SPARK-43183
                 URL: https://issues.apache.org/jira/browse/SPARK-43183
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.5.0
            Reporter: Jungtaek Lim


People has been having a lot of confusions about update event on idleness; it’s not only the matter of understanding but also comes up with various types of complaints. For example, since we give the latest batch ID for update event on idleness, if the listener implementation blindly performs upsert based on batch ID, they are in risk to lose metrics.

This also complicates the logic because we have to memorize the execution for the previous batch, which is arguably not necessary.

Because of this, we’d be better to move the idle event out of progress update event and have separate callback method for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org