[jira] [Resolved] (SPARK-37970) Introduce a new interface on streaming data source to notify the latest seen offset

["Yuanjian Li (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-37970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yuanjian Li resolved SPARK-37970.
---------------------------------
    Fix Version/s: 3.3.0
       Resolution: Fixed

Issue resolved by pull request 35259
[https://github.com/apache/spark/pull/35259]

> Introduce a new interface on streaming data source to notify the latest seen offset
> -----------------------------------------------------------------------------------
>
>                 Key: SPARK-37970
>                 URL: https://issues.apache.org/jira/browse/SPARK-37970
>             Project: Spark
>          Issue Type: New Feature
>          Components: Structured Streaming
>    Affects Versions: 3.3.0
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Major
>             Fix For: 3.3.0
>
>
> We figure out the case of streaming data source that knowing the latest seen offset when restarting query would be handy and useful to implement some feature. One useful case is enabling the data source to track the offset by itself, for the case where the external storage of data source is not exposing any API to provide the latest available offset.
> We will propose a new interface on streaming data source, which indicates Spark to give the latest seen offset whenever the query is being restarted. For the first start of the query, the initial offset of the data source should be retrieved from calling initialOffset.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org