[jira] [Resolved] (SPARK-43183) Move update event on idleness in streaming query listener to separate callback method

["Jungtaek Lim (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-43183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim resolved SPARK-43183.
----------------------------------
    Fix Version/s: 3.5.0
       Resolution: Fixed

Issue resolved by pull request 40845
[https://github.com/apache/spark/pull/40845]

> Move update event on idleness in streaming query listener to separate callback method
> -------------------------------------------------------------------------------------
>
>                 Key: SPARK-43183
>                 URL: https://issues.apache.org/jira/browse/SPARK-43183
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.5.0
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Major
>             Fix For: 3.5.0
>
>
> People has been having a lot of confusions about update event on idleness; it’s not only the matter of understanding but also comes up with various types of complaints. For example, since we give the latest batch ID for update event on idleness, if the listener implementation blindly performs upsert based on batch ID, they are in risk to lose metrics.
> This also complicates the logic because we have to memorize the execution for the previous batch, which is arguably not necessary.
> Because of this, we’d be better to move the idle event out of progress update event and have separate callback method for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org