[jira] [Created] (SPARK-26848) Introduce new option to Kafka source - specify timestamp to start and end offset

["Jungtaek Lim (JIRA)" <ji...@apache.org>]

Jungtaek Lim created SPARK-26848:
------------------------------------

             Summary: Introduce new option to Kafka source - specify timestamp to start and end offset
                 Key: SPARK-26848
                 URL: https://issues.apache.org/jira/browse/SPARK-26848
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Jungtaek Lim


Kafka source (for SQL/SS) provides options to set specific offset per topic partition to let source starts reading from start offsets and ends reading until end offsets. ("startingOffsets" and "endingOffsets" in below document.)

http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html

I'd like to introduce new options "startingOffsetsByTimestamp" and "endingOffsetsByTimestamp" to set specific timestamp per topic (since we're unlikely to set the different value per partition) to let source starts reading from offsets which have equal of greater timestamp, and ends reading until offsets which have equal of greater timestamp.

The option would be optional of course, and have a preference to apply the options 1) timestamp option applies first, and if it doesn't exist, offset option applies.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org