You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2019/02/08 13:51:00 UTC
[jira] [Created] (SPARK-26849) Introduce new option to Kafka
source: offset by timestamp (starting/ending)
Jungtaek Lim created SPARK-26849:
------------------------------------
Summary: Introduce new option to Kafka source: offset by timestamp (starting/ending)
Key: SPARK-26849
URL: https://issues.apache.org/jira/browse/SPARK-26849
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: Jungtaek Lim
Now Kafka source provides options to specify custom offset per topic partition to set where to start reading from, and where to stop reading.
I'd like to introduce new options to specify timestamp per topic (not topic partition - we can support but it would be unlikely for us to set timestamp per partition) to fetch offset via timestamp and start reading from and stop reading.
The characteristic of new options would be very similar to existing options. For example, in streaming query, ending timestamp option would not be valid, and starting timestamp option only affects when query starts - if query restores from checkpoint the option would not be in effect.
The new timestamp option would take precedence over offset option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org