You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2019/02/08 13:55:00 UTC
[jira] [Resolved] (SPARK-26849) Introduce new option to Kafka
source: offset by timestamp (starting/ending)
[ https://issues.apache.org/jira/browse/SPARK-26849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-26849.
----------------------------------
Resolution: Invalid
Didn't notice SPARK-26848 is created. Closing.
> Introduce new option to Kafka source: offset by timestamp (starting/ending)
> ---------------------------------------------------------------------------
>
> Key: SPARK-26849
> URL: https://issues.apache.org/jira/browse/SPARK-26849
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> Now Kafka source provides options to specify custom offset per topic partition to set where to start reading from, and where to stop reading.
> I'd like to introduce new options to specify timestamp per topic (not topic partition - we can support but it would be unlikely for us to set timestamp per partition) to fetch offset via timestamp and start reading from and stop reading.
> The characteristic of new options would be very similar to existing options. For example, in streaming query, ending timestamp option would not be valid, and starting timestamp option only affects when query starts - if query restores from checkpoint the option would not be in effect.
> The new timestamp option would take precedence over offset option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org