[jira] [Comment Edited] (NIFI-4985) Allow users to define a specific offset when starting ConsumeKafka

["Dennis Jaheruddin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242361#comment-17242361 ] 

Dennis Jaheruddin edited comment on NIFI-4985 at 12/2/20, 1:19 PM:
-------------------------------------------------------------------

I agree that the suggestion in its original form is most valuable. However, perhaps it would make sense to start with something more lightweight that would still solve the case 'we need to re-load because we did something wrong but we can't afford to start from the beginning and calling the Kafka team takes forever'.

If we would be able to make this work for users without exact knowledge of how the data is stored in Kafka, this would be even more convenient.

Some thoughts:
 * Rather than specifying a specific offset, allow something like 'reset to read last x messages' (presumably per partition).
 * Or perhaps even simpler would be 'reset to timestamp Y'

This would logically still be defined, even if you have multiple topics and/or partitions.

 


was (Author: dennisjaheruddin):
I agree that the suggestion in its original form is most valuable. However, perhaps it would make sense to start with something more lightweight that would still solve the case 'we need to re-load because we did something wrong'.

If we would be able to make this work for users without exact knowledge of how the data is stored in Kafka, this would be even more convenient.

Some thoughts:
 * Rather than specifying a specific offset, allow something like 'reset to read last x messages' (presumably per partition).
 * Or perhaps even simpler would be 'reset to timestamp Y'

This would logically still be defined, even if you have multiple topics and/or partitions.

 

> Allow users to define a specific offset when starting ConsumeKafka
> ------------------------------------------------------------------
>
>                 Key: NIFI-4985
>                 URL: https://issues.apache.org/jira/browse/NIFI-4985
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Pierre Villard
>            Assignee: Sandish Kumar HN
>            Priority: Major
>
> It'd be useful to add support for dynamic properties in ConsumeKafka set of processors so that users can define the offset to use when starting the processor. The properties could be something like:
> {noformat}
> kafka.<topic>.<partition>.offset{noformat}
> If, for a configured topic, such a property is not defined for a given partition, the consumer would use the auto offset property.
> If a custom offset is defined for a topic/partition, it'd be used when initializing the consumer by calling:
> {noformat}
> seek(TopicPartition, long){noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)