[jira] [Commented] (NIFI-4985) Allow users to define a specific offset when starting ConsumeKafka

["Dennis Jaheruddin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242361#comment-17242361 ] 

Dennis Jaheruddin commented on NIFI-4985:
-----------------------------------------

I agree that the suggestion in its original form is most valuable. However, perhaps it would make sense to start with something more lightweight that would still solve the case 'we need to re-load because we did something wrong'.

If we would be able to make this work for users without exact knowledge of how the data is stored in Kafka, this would be even more convenient.

Some thoughts:
 * Rather than specifying a specific offset, allow something like 'reset to read last x messages' (presumably per partition).
 * Or perhaps even simpler would be 'reset to timestamp'

 

> Allow users to define a specific offset when starting ConsumeKafka
> ------------------------------------------------------------------
>
>                 Key: NIFI-4985
>                 URL: https://issues.apache.org/jira/browse/NIFI-4985
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Pierre Villard
>            Assignee: Sandish Kumar HN
>            Priority: Major
>
> It'd be useful to add support for dynamic properties in ConsumeKafka set of processors so that users can define the offset to use when starting the processor. The properties could be something like:
> {noformat}
> kafka.<topic>.<partition>.offset{noformat}
> If, for a configured topic, such a property is not defined for a given partition, the consumer would use the auto offset property.
> If a custom offset is defined for a topic/partition, it'd be used when initializing the consumer by calling:
> {noformat}
> seek(TopicPartition, long){noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)