Resend old messages, find partitions for a message

["Greer, Andrew C" <An...@conocophillips.com>]

Hello all,

I am looking to add a "resend" option to my program where a user can specify an older message they would like to produce through Kafka again, for whatever reason. I can get the topic and offset for each message from my logs after consuming a message, but I do not see a way to get which partition the message was in. Currently, I do not have a key associated with the messages.

My thoughts right now lean towards adding a key system to know which partition each message will go to, but I was wondering if there are better alternatives to this.


Thank you,

Andrew Greer

Re: Resend old messages, find partitions for a message

[Ryanne Dolan <ry...@gmail.com>]

Andrew, an offset implies a partition -- an offset is only meaningful
within the context of a particular partition -- so if you are able to log
offsets you should also be able to log the corresponding partitions.

For example, the RecordMetadata object, which provides the offset of a
written record, also includes the corresponding TopicPartition.

Ryanne

On Mon, Jun 14, 2021, 11:14 AM Greer, Andrew C <
Andrew.C.Greer@conocophillips.com> wrote:

> Hello all,
>
> I am looking to add a "resend" option to my program where a user can
> specify an older message they would like to produce through Kafka again,
> for whatever reason. I can get the topic and offset for each message from
> my logs after consuming a message, but I do not see a way to get which
> partition the message was in. Currently, I do not have a key associated
> with the messages.
>
> My thoughts right now lean towards adding a key system to know which
> partition each message will go to, but I was wondering if there are better
> alternatives to this.
>
>
> Thank you,
>
> Andrew Greer
>
>