[jira] [Created] (SPARK-37970) Introduce a new interface on streaming data source to notify the latest seen offset

["Jungtaek Lim (Jira)" <ji...@apache.org>]

Jungtaek Lim created SPARK-37970:
------------------------------------

             Summary: Introduce a new interface on streaming data source to notify the latest seen offset
                 Key: SPARK-37970
                 URL: https://issues.apache.org/jira/browse/SPARK-37970
             Project: Spark
          Issue Type: New Feature
          Components: Structured Streaming
    Affects Versions: 3.3.0
            Reporter: Jungtaek Lim


We figure out the case of streaming data source that knowing the latest seen offset when restarting query would be handy and useful to implement some feature. One useful case is enabling the data source to track the offset by itself, for the case where the external storage of data source is not exposing any API to provide the latest available offset.

We will propose a new interface on streaming data source, which indicates Spark to give the latest seen offset whenever the query is being restarted. For the first start of the query, the initial offset of the data source should be retrieved from calling initialOffset.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org