Confluent's parallel consumer

[Pushkar Deole <pd...@gmail.com>]

Hi All, and Antony (author of below article)

i came across this article which seemed interesting: Introducing
Confluent’s Parallel Consumer Message Processing Client
<https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/>

I would like to use the key-level ordering strategy mentioned in the
article to scale my consumers, however I would like to check how the offset
commits are handled in this strategy
e.g. on partition 1, key1 has offsets 20 and 30 respectively and on the
same partition key2 has offset 40. With key-level ordering model, key2 will
be processed by a different thread in background and might gets processed
before events related to key1, in this case offset for key2 will be
committed before key1 gets processed ? How is this handled?

Re: Confluent's parallel consumer

[Antony Stubbs <an...@confluent.io.INVALID>]

Hi Israel, Pushkar, everyone! Sorry I didn't notice this message. Feel
free to collaborate on the github discussions page about the project too.

And yes Israel is correct - offsets are handled correctly regardless of
what modes you choose. Offsets are tracked individually, and committed to
kafka inside the commit metadata payload - there is no message loss.
Here's a conversation on the topic in the discussion forum:
https://github.com/confluentinc/parallel-consumer/discussions/135#discussioncomment-1050437

Let me know if I can help with anything else!

Cheers,
Antony.

On Thu, Jul 15, 2021 at 11:50 PM Israel Ekpo <is...@gmail.com> wrote:

> Hi Pushkar,
>
> Based on what I understand about the library, I don't think you need to
> worry about data loss because there are mechanisms in place to track which
> offsets have been processed in the event something goes wrong during
> processing.
>
> If the processing client goes offline or is unresponsive and has to be
> resumed or restarted, it should restart from where it left off since the
> offsets are being tracked.
>
>
> https://www.confluent.io/events/kafka-summit-europe-2021/introducing-confluent-labs-parallel-consumer-client/
>
> This should be true regardless of the ordering strategy you select.
>
> Take a look at the talk from the most recent Kafka Summit for additional
> details on this
>
> The github documentation for the project also covers this briefly
>
>
> https://github.com/confluentinc/parallel-consumer/blob/master/README.adoc#commit-mode
>
> I hope this was helpful. If you still have additional questions, please
> continue to bring it up.
>
> Thanks.
>
> On Thu, Jul 15, 2021 at 11:21 AM Pushkar Deole <pd...@gmail.com>
> wrote:
>
> > It is consumer client node that has received events and is processing
> > those...
> >
> > On Thu, Jul 15, 2021 at 8:49 PM Israel Ekpo <is...@gmail.com>
> wrote:
> >
> > > Hi Pushkar,
> > >
> > > When you use the term “node/instance” are you referring to the Kafka
> > > Brokers or the consuming clients that are retrieving events from the
> > > broker?
> > >
> > > Please could you elaborate/clarify?
> > >
> > > On Thu, Jul 15, 2021 at 10:00 AM Pushkar Deole <pd...@gmail.com>
> > > wrote:
> > >
> > > > Well... with key-level ordering, i am mainly concerned about event
> > loss,
> > > if
> > > > any, in below mentioned scenario:
> > > >
> > > > 1. since event1 with key1 and event2 with key2 are both part of the
> > same
> > > > partition1
> > > > 2. key1 event has offset 30 while key2 has offset 40
> > > > 3. key2 is processed by background thread and offset committed which
> is
> > > 40
> > > > 4. before key1 gets processed by background thread, the instance/node
> > > goes
> > > > down
> > > > 5. partition1 gets rebalanced to node2 and start processing ahead of
> > > offset
> > > > 40, thus losing key1
> > > >
> > > >
> > > >
> > > > On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo <is...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Pushkar,
> > > > >
> > > > > If you are selecting key-based ordering, you should not be
> concerned
> > > > about
> > > > > the other keys from the same partitions being committed first
> > > > >
> > > > > If that is a concern for your use cases then you should go with
> > > partition
> > > > > based ordering to ensure that the events are processed in the
> > sequence
> > > > they
> > > > > are picked up from the topic partition.
> > > > >
> > > > > For commit mode, you have the asynchronous, synchronous and
> > > transactional
> > > > > modes. I think if you are concerned with the order of commits you
> > > should
> > > > > look into the last two modes.
> > > > >
> > > > > My recommendation would be to go with the partition based ordering
> > with
> > > > > synchronous commits to start with.
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole <
> pdeole2015@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi All, and Antony (author of below article)
> > > > > >
> > > > > > i came across this article which seemed interesting: Introducing
> > > > > > Confluent’s Parallel Consumer Message Processing Client
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/
> > > > > > >
> > > > > >
> > > > > > I would like to use the key-level ordering strategy mentioned in
> > the
> > > > > > article to scale my consumers, however I would like to check how
> > the
> > > > > offset
> > > > > > commits are handled in this strategy
> > > > > > e.g. on partition 1, key1 has offsets 20 and 30 respectively and
> on
> > > the
> > > > > > same partition key2 has offset 40. With key-level ordering model,
> > > key2
> > > > > will
> > > > > > be processed by a different thread in background and might gets
> > > > processed
> > > > > > before events related to key1, in this case offset for key2 will
> be
> > > > > > committed before key1 gets processed ? How is this handled?
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 

<https://www.confluent.io>

Antony Stubbs


Follow us:  Blog
<https://confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>
• Slack <https://slackpass.io/confluentcommunity> • Twitter
<https://twitter.com/ConfluentInc> • YouTube <https://youtube.com/confluent>

Re: Confluent's parallel consumer

[Israel Ekpo <is...@gmail.com>]

Hi Pushkar,

Based on what I understand about the library, I don't think you need to
worry about data loss because there are mechanisms in place to track which
offsets have been processed in the event something goes wrong during
processing.

If the processing client goes offline or is unresponsive and has to be
resumed or restarted, it should restart from where it left off since the
offsets are being tracked.

https://www.confluent.io/events/kafka-summit-europe-2021/introducing-confluent-labs-parallel-consumer-client/

This should be true regardless of the ordering strategy you select.

Take a look at the talk from the most recent Kafka Summit for additional
details on this

The github documentation for the project also covers this briefly

https://github.com/confluentinc/parallel-consumer/blob/master/README.adoc#commit-mode

I hope this was helpful. If you still have additional questions, please
continue to bring it up.

Thanks.

On Thu, Jul 15, 2021 at 11:21 AM Pushkar Deole <pd...@gmail.com> wrote:

> It is consumer client node that has received events and is processing
> those...
>
> On Thu, Jul 15, 2021 at 8:49 PM Israel Ekpo <is...@gmail.com> wrote:
>
> > Hi Pushkar,
> >
> > When you use the term “node/instance” are you referring to the Kafka
> > Brokers or the consuming clients that are retrieving events from the
> > broker?
> >
> > Please could you elaborate/clarify?
> >
> > On Thu, Jul 15, 2021 at 10:00 AM Pushkar Deole <pd...@gmail.com>
> > wrote:
> >
> > > Well... with key-level ordering, i am mainly concerned about event
> loss,
> > if
> > > any, in below mentioned scenario:
> > >
> > > 1. since event1 with key1 and event2 with key2 are both part of the
> same
> > > partition1
> > > 2. key1 event has offset 30 while key2 has offset 40
> > > 3. key2 is processed by background thread and offset committed which is
> > 40
> > > 4. before key1 gets processed by background thread, the instance/node
> > goes
> > > down
> > > 5. partition1 gets rebalanced to node2 and start processing ahead of
> > offset
> > > 40, thus losing key1
> > >
> > >
> > >
> > > On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo <is...@gmail.com>
> > wrote:
> > >
> > > > Hi Pushkar,
> > > >
> > > > If you are selecting key-based ordering, you should not be concerned
> > > about
> > > > the other keys from the same partitions being committed first
> > > >
> > > > If that is a concern for your use cases then you should go with
> > partition
> > > > based ordering to ensure that the events are processed in the
> sequence
> > > they
> > > > are picked up from the topic partition.
> > > >
> > > > For commit mode, you have the asynchronous, synchronous and
> > transactional
> > > > modes. I think if you are concerned with the order of commits you
> > should
> > > > look into the last two modes.
> > > >
> > > > My recommendation would be to go with the partition based ordering
> with
> > > > synchronous commits to start with.
> > > >
> > > >
> > > >
> > > > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole <pd...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All, and Antony (author of below article)
> > > > >
> > > > > i came across this article which seemed interesting: Introducing
> > > > > Confluent’s Parallel Consumer Message Processing Client
> > > > > <
> > > > >
> > > >
> > >
> >
> https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/
> > > > > >
> > > > >
> > > > > I would like to use the key-level ordering strategy mentioned in
> the
> > > > > article to scale my consumers, however I would like to check how
> the
> > > > offset
> > > > > commits are handled in this strategy
> > > > > e.g. on partition 1, key1 has offsets 20 and 30 respectively and on
> > the
> > > > > same partition key2 has offset 40. With key-level ordering model,
> > key2
> > > > will
> > > > > be processed by a different thread in background and might gets
> > > processed
> > > > > before events related to key1, in this case offset for key2 will be
> > > > > committed before key1 gets processed ? How is this handled?
> > > > >
> > > >
> > >
> >
>

Re: Confluent's parallel consumer

[Pushkar Deole <pd...@gmail.com>]

It is consumer client node that has received events and is processing
those...

On Thu, Jul 15, 2021 at 8:49 PM Israel Ekpo <is...@gmail.com> wrote:

> Hi Pushkar,
>
> When you use the term “node/instance” are you referring to the Kafka
> Brokers or the consuming clients that are retrieving events from the
> broker?
>
> Please could you elaborate/clarify?
>
> On Thu, Jul 15, 2021 at 10:00 AM Pushkar Deole <pd...@gmail.com>
> wrote:
>
> > Well... with key-level ordering, i am mainly concerned about event loss,
> if
> > any, in below mentioned scenario:
> >
> > 1. since event1 with key1 and event2 with key2 are both part of the same
> > partition1
> > 2. key1 event has offset 30 while key2 has offset 40
> > 3. key2 is processed by background thread and offset committed which is
> 40
> > 4. before key1 gets processed by background thread, the instance/node
> goes
> > down
> > 5. partition1 gets rebalanced to node2 and start processing ahead of
> offset
> > 40, thus losing key1
> >
> >
> >
> > On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo <is...@gmail.com>
> wrote:
> >
> > > Hi Pushkar,
> > >
> > > If you are selecting key-based ordering, you should not be concerned
> > about
> > > the other keys from the same partitions being committed first
> > >
> > > If that is a concern for your use cases then you should go with
> partition
> > > based ordering to ensure that the events are processed in the sequence
> > they
> > > are picked up from the topic partition.
> > >
> > > For commit mode, you have the asynchronous, synchronous and
> transactional
> > > modes. I think if you are concerned with the order of commits you
> should
> > > look into the last two modes.
> > >
> > > My recommendation would be to go with the partition based ordering with
> > > synchronous commits to start with.
> > >
> > >
> > >
> > > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole <pd...@gmail.com>
> > > wrote:
> > >
> > > > Hi All, and Antony (author of below article)
> > > >
> > > > i came across this article which seemed interesting: Introducing
> > > > Confluent’s Parallel Consumer Message Processing Client
> > > > <
> > > >
> > >
> >
> https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/
> > > > >
> > > >
> > > > I would like to use the key-level ordering strategy mentioned in the
> > > > article to scale my consumers, however I would like to check how the
> > > offset
> > > > commits are handled in this strategy
> > > > e.g. on partition 1, key1 has offsets 20 and 30 respectively and on
> the
> > > > same partition key2 has offset 40. With key-level ordering model,
> key2
> > > will
> > > > be processed by a different thread in background and might gets
> > processed
> > > > before events related to key1, in this case offset for key2 will be
> > > > committed before key1 gets processed ? How is this handled?
> > > >
> > >
> >
>

Re: Confluent's parallel consumer

[Israel Ekpo <is...@gmail.com>]

Hi Pushkar,

When you use the term “node/instance” are you referring to the Kafka
Brokers or the consuming clients that are retrieving events from the broker?

Please could you elaborate/clarify?

On Thu, Jul 15, 2021 at 10:00 AM Pushkar Deole <pd...@gmail.com> wrote:

> Well... with key-level ordering, i am mainly concerned about event loss, if
> any, in below mentioned scenario:
>
> 1. since event1 with key1 and event2 with key2 are both part of the same
> partition1
> 2. key1 event has offset 30 while key2 has offset 40
> 3. key2 is processed by background thread and offset committed which is 40
> 4. before key1 gets processed by background thread, the instance/node goes
> down
> 5. partition1 gets rebalanced to node2 and start processing ahead of offset
> 40, thus losing key1
>
>
>
> On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo <is...@gmail.com> wrote:
>
> > Hi Pushkar,
> >
> > If you are selecting key-based ordering, you should not be concerned
> about
> > the other keys from the same partitions being committed first
> >
> > If that is a concern for your use cases then you should go with partition
> > based ordering to ensure that the events are processed in the sequence
> they
> > are picked up from the topic partition.
> >
> > For commit mode, you have the asynchronous, synchronous and transactional
> > modes. I think if you are concerned with the order of commits you should
> > look into the last two modes.
> >
> > My recommendation would be to go with the partition based ordering with
> > synchronous commits to start with.
> >
> >
> >
> > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole <pd...@gmail.com>
> > wrote:
> >
> > > Hi All, and Antony (author of below article)
> > >
> > > i came across this article which seemed interesting: Introducing
> > > Confluent’s Parallel Consumer Message Processing Client
> > > <
> > >
> >
> https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/
> > > >
> > >
> > > I would like to use the key-level ordering strategy mentioned in the
> > > article to scale my consumers, however I would like to check how the
> > offset
> > > commits are handled in this strategy
> > > e.g. on partition 1, key1 has offsets 20 and 30 respectively and on the
> > > same partition key2 has offset 40. With key-level ordering model, key2
> > will
> > > be processed by a different thread in background and might gets
> processed
> > > before events related to key1, in this case offset for key2 will be
> > > committed before key1 gets processed ? How is this handled?
> > >
> >
>

Re: Confluent's parallel consumer

[Pushkar Deole <pd...@gmail.com>]

Well... with key-level ordering, i am mainly concerned about event loss, if
any, in below mentioned scenario:

1. since event1 with key1 and event2 with key2 are both part of the same
partition1
2. key1 event has offset 30 while key2 has offset 40
3. key2 is processed by background thread and offset committed which is 40
4. before key1 gets processed by background thread, the instance/node goes
down
5. partition1 gets rebalanced to node2 and start processing ahead of offset
40, thus losing key1



On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo <is...@gmail.com> wrote:

> Hi Pushkar,
>
> If you are selecting key-based ordering, you should not be concerned about
> the other keys from the same partitions being committed first
>
> If that is a concern for your use cases then you should go with partition
> based ordering to ensure that the events are processed in the sequence they
> are picked up from the topic partition.
>
> For commit mode, you have the asynchronous, synchronous and transactional
> modes. I think if you are concerned with the order of commits you should
> look into the last two modes.
>
> My recommendation would be to go with the partition based ordering with
> synchronous commits to start with.
>
>
>
> On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole <pd...@gmail.com>
> wrote:
>
> > Hi All, and Antony (author of below article)
> >
> > i came across this article which seemed interesting: Introducing
> > Confluent’s Parallel Consumer Message Processing Client
> > <
> >
> https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/
> > >
> >
> > I would like to use the key-level ordering strategy mentioned in the
> > article to scale my consumers, however I would like to check how the
> offset
> > commits are handled in this strategy
> > e.g. on partition 1, key1 has offsets 20 and 30 respectively and on the
> > same partition key2 has offset 40. With key-level ordering model, key2
> will
> > be processed by a different thread in background and might gets processed
> > before events related to key1, in this case offset for key2 will be
> > committed before key1 gets processed ? How is this handled?
> >
>

Re: Confluent's parallel consumer

[Israel Ekpo <is...@gmail.com>]

Hi Pushkar,

If you are selecting key-based ordering, you should not be concerned about
the other keys from the same partitions being committed first

If that is a concern for your use cases then you should go with partition
based ordering to ensure that the events are processed in the sequence they
are picked up from the topic partition.

For commit mode, you have the asynchronous, synchronous and transactional
modes. I think if you are concerned with the order of commits you should
look into the last two modes.

My recommendation would be to go with the partition based ordering with
synchronous commits to start with.



On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole <pd...@gmail.com> wrote:

> Hi All, and Antony (author of below article)
>
> i came across this article which seemed interesting: Introducing
> Confluent’s Parallel Consumer Message Processing Client
> <
> https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/
> >
>
> I would like to use the key-level ordering strategy mentioned in the
> article to scale my consumers, however I would like to check how the offset
> commits are handled in this strategy
> e.g. on partition 1, key1 has offsets 20 and 30 respectively and on the
> same partition key2 has offset 40. With key-level ordering model, key2 will
> be processed by a different thread in background and might gets processed
> before events related to key1, in this case offset for key2 will be
> committed before key1 gets processed ? How is this handled?
>