RdKafka & setting offsets

["Doyle, Keith" <Ke...@greenwayhealth.com>]

I'm attempting to set the offset for a RdKafka-dotnet consumer in order to re-read the topic, but I've not seen any documentation or examples that do this.   I saw a reference for librdkafka that seems to show a set_offset method off the TopicPartition class and an Offset property on the TopicPartitionOffset class, but instantiating the former in RdKafka-dotnet shows no such method, so I'm not sure it's been extended to RdKafka-dotnet.   In addition, I have no idea how to go about using these TopicPartition/Offset classes in the context of the RdKafka EventConsumer, whether it should be passed in as a parameter to the Subscribe or Start methods, or to the Position method (and if so, when to invoke it-before or after the Subscribe & Start, etc...).

Can anyone clarify how you're supposed to set the offset on a topic in the context of RdKafka?

--

[Greenway_Health_PNG_large_NO_tag]

Keith Doyle  |  Senior Software Engineer
Greenway Health  |  4301 W. Boy Scout Blvd., Suite 800, Tampa, FL 33607
(702) 256-9911 office  |  GreenwayHealth.com<http://www.greenwayhealth.com/>
[cid:image003.png@01D04086.868CBCB0]<https://www.linkedin.com/company/greenway-health>[cid:image004.png@01D04086.868CBCB0]<https://www.facebook.com/GreenwayHealthLLC>[cid:image005.png@01D04086.868CBCB0]<https://twitter.com/greenway>

NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged and confidential information intended solely for the use of the addressee. If the reader of this message is not the intended recipient, you are hereby notified that any reading, dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by electronic mail and delete this message and all copies and backups thereof. Thank you. Greenway Health.

Re: RdKafka & setting offsets

[Matt Howlett <ma...@confluent.io>]

Two ideas:

you could use a new consumer group id and set the TopicConfig property
"auto.offset.reset" to "smallest". Consumers in the new group will read
from the beginning on all partitions.

Alternatively, as an example of how to set the offset explicitly, you can
modify the AdvancedConsumer example in rdkafka-dotnet by changing line 62
to:
consumer.Assign(partitions.Select(p => new TopicPartitionOffset(p.Topic,
P.Partition, 0)).ToList());
the consumer will read from offset 0 on all partitions.

Note how this example works: It first subscribes to a topic using the
.Subscribe method on EventConsumer. It then receives a partition allocation
from Kafka via the OnPartitionAssigned event. In order to actually start
consuming any messages, you must explicitly call the Assign method with a
list of TopicPartitionOffsets. You will typically call this with the list
passed to you in the OnPartitionAssigned event unmodified, but not
necessarily. Here we chose to explicitly set the offset from which to start
reading.

Side note (to aid understanding): it's is also possible to not use Kafka's
built in group management and manage partition allocation yourself. To do
this, you'd not call .Subscribe, not implement OnPartitionAssigned and
OnPartitionRevoked and call the Assign method with your desired
TopicPartitionOffset's before calling consumer.Start().



On Wed, Dec 7, 2016 at 12:53 PM, Doyle, Keith <
Keith.Doyle@greenwayhealth.com> wrote:

>
>
> I’m attempting to set the offset for a RdKafka-dotnet consumer in order to
> re-read the topic, but I’ve not seen any documentation or examples that do
> this.   I saw a reference for librdkafka that seems to show a set_offset
> method off the TopicPartition class and an Offset property on the
> TopicPartitionOffset class, but instantiating the former in RdKafka-dotnet
> shows no such method, so I’m not sure it’s been extended to
> RdKafka-dotnet.   In addition, I have no idea how to go about using these
> TopicPartition/Offset classes in the context of the RdKafka EventConsumer,
> whether it should be passed in as a parameter to the Subscribe or Start
> methods, or to the Position method (and if so, when to invoke it—before or
> after the Subscribe & Start, etc…).
>
>
>
> Can anyone clarify how you’re supposed to set the offset on a topic in the
> context of RdKafka?
>
>
>
> --
>
>
>
> [image: Greenway_Health_PNG_large_NO_tag]
>
>
>
> *Keith Doyle*  *|*  Senior Software Engineer
>
> *Greenway Health*  *|*  4301 W. Boy Scout Blvd., Suite 800, Tampa, FL
> 33607
>
> (702) 256-9911 office  *|*  GreenwayHealth.com
> <http://www.greenwayhealth.com/>
>
> [image: cid:image003.png@01D04086.868CBCB0]
> <https://www.linkedin.com/company/greenway-health>[image:
> cid:image004.png@01D04086.868CBCB0]
> <https://www.facebook.com/GreenwayHealthLLC>[image:
> cid:image005.png@01D04086.868CBCB0] <https://twitter.com/greenway>
>
>
> NOTICE: This e-mail message and all attachments transmitted with it may
> contain legally privileged and confidential information intended solely for
> the use of the addressee. If the reader of this message is not the intended
> recipient, you are hereby notified that any reading, dissemination,
> distribution, copying, or other use of this message or its attachments is
> strictly prohibited. If you have received this message in error, please
> notify the sender immediately by electronic mail and delete this message
> and all copies and backups thereof. Thank you. Greenway Health.
>