[jira] [Assigned] (SPARK-26848) Introduce new option to Kafka source - specify timestamp to start and end offset

["Sean Owen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-26848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen reassigned SPARK-26848:
---------------------------------

    Assignee: Jungtaek Lim

> Introduce new option to Kafka source - specify timestamp to start and end offset
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-26848
>                 URL: https://issues.apache.org/jira/browse/SPARK-26848
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Major
>
> Kafka source (for SQL/SS) provides options to set specific offset per topic partition to let source starts reading from start offsets and ends reading until end offsets. ("startingOffsets" and "endingOffsets" in below document.)
> http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html
> I'd like to introduce new options "startingOffsetsByTimestamp" and "endingOffsetsByTimestamp" to set specific timestamp per topic (since we're unlikely to set the different value per partition) to let source starts reading from offsets which have equal of greater timestamp, and ends reading until offsets which have equal of greater timestamp.
> The option would be optional of course, and have a preference to apply the options 1) timestamp option applies first, and if it doesn't exist, offset option applies.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org