You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sree Sanjeev Kandasamy Gokulrajan <sr...@gmail.com> on 2023/10/03 14:10:31 UTC
Multiple consumers for a Single Partition in Kafka
I'm interested in knowing if we can achieve parallelism by allowing
multiple consumers to subscribe to a single partition of a topic.
To explore potential solutions, I'm considering the following approach.
1. Implementing a locking mechanism to control access to the offsets in a
partition to avoid concurrent processing issues.
My primary concern is maintaining message ordering within the partition, as
Kafka guarantees ordering within a partition. and message ordering is not
the primary concern for many applications, so can we have a flag to enable
multiple consumers to a single partition ?
I appreciate any insights, advice, or experiences that the Kafka community
can share on this topic. Thank you!
Regards,
Sree Sanjeev.
Re: Multiple consumers for a Single Partition in Kafka
Posted by Alex Craig <al...@gmail.com>.
Sounds like the parallel consumer is what you want:
https://github.com/confluentinc/parallel-consumer
Or at least decide why that won't work before writing something yourself.
- alex
On Tue, Oct 3, 2023 at 4:20 PM Sree Sanjeev Kandasamy Gokulrajan <
sreesanjeevkg@gmail.com> wrote:
> Hey Neeraj
> I had a kafka assignment where I had to consume from a single partition.
> When I was looking to scale the application, I was planning to consume the
> partition with more than one consumer when I came across this issue.
>
> Regards,
> Sree Sanjeev.
>
> On Tue, Oct 3, 2023 at 3:48 PM Neeraj Vaidya
> <ne...@yahoo.co.in.invalid> wrote:
>
> > Hi Sree Sanjeev,
> > Other than the info from Colt, why can't you just increase partitions and
> > scale the number of consumers accordingly ?
> > Or build your application as a KStreams application and take advantage of
> > horizontal and vertical scalability ?
> >
> > Regards,
> > Neeraj
> >
> >
> > On Wednesday, 4 October, 2023 at 01:19:37 am GMT+11, Colt McNealy <
> > colt@littlehorse.io> wrote:
> >
> > Hello,
> >
> > Currently, you cannot have more than one consumer consuming from a
> > partition in Kafka. However, there is active discussion about a proposal
> to
> > enable queueing semantics for Kafka, which is a superset of the feature
> you
> > are talking about. Feel free to check KIP-932:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> >
> > Hope this helps,
> > Colt McNealy
> >
> > *Founder, LittleHorse.dev*
> >
> >
> > On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan <
> > sreesanjeevkg@gmail.com> wrote:
> >
> > > I'm interested in knowing if we can achieve parallelism by allowing
> > > multiple consumers to subscribe to a single partition of a topic.
> > >
> > > To explore potential solutions, I'm considering the following approach.
> > >
> > > 1. Implementing a locking mechanism to control access to the offsets
> in a
> > > partition to avoid concurrent processing issues.
> > >
> > > My primary concern is maintaining message ordering within the
> partition,
> > as
> > > Kafka guarantees ordering within a partition. and message ordering is
> not
> > > the primary concern for many applications, so can we have a flag to
> > enable
> > > multiple consumers to a single partition ?
> > >
> > > I appreciate any insights, advice, or experiences that the Kafka
> > community
> > > can share on this topic. Thank you!
> > >
> > > Regards,
> > > Sree Sanjeev.
> > >
> >
>
Re: Multiple consumers for a Single Partition in Kafka
Posted by Sree Sanjeev Kandasamy Gokulrajan <sr...@gmail.com>.
Hey Neeraj
I had a kafka assignment where I had to consume from a single partition.
When I was looking to scale the application, I was planning to consume the
partition with more than one consumer when I came across this issue.
Regards,
Sree Sanjeev.
On Tue, Oct 3, 2023 at 3:48 PM Neeraj Vaidya
<ne...@yahoo.co.in.invalid> wrote:
> Hi Sree Sanjeev,
> Other than the info from Colt, why can't you just increase partitions and
> scale the number of consumers accordingly ?
> Or build your application as a KStreams application and take advantage of
> horizontal and vertical scalability ?
>
> Regards,
> Neeraj
>
>
> On Wednesday, 4 October, 2023 at 01:19:37 am GMT+11, Colt McNealy <
> colt@littlehorse.io> wrote:
>
> Hello,
>
> Currently, you cannot have more than one consumer consuming from a
> partition in Kafka. However, there is active discussion about a proposal to
> enable queueing semantics for Kafka, which is a superset of the feature you
> are talking about. Feel free to check KIP-932:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>
> Hope this helps,
> Colt McNealy
>
> *Founder, LittleHorse.dev*
>
>
> On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan <
> sreesanjeevkg@gmail.com> wrote:
>
> > I'm interested in knowing if we can achieve parallelism by allowing
> > multiple consumers to subscribe to a single partition of a topic.
> >
> > To explore potential solutions, I'm considering the following approach.
> >
> > 1. Implementing a locking mechanism to control access to the offsets in a
> > partition to avoid concurrent processing issues.
> >
> > My primary concern is maintaining message ordering within the partition,
> as
> > Kafka guarantees ordering within a partition. and message ordering is not
> > the primary concern for many applications, so can we have a flag to
> enable
> > multiple consumers to a single partition ?
> >
> > I appreciate any insights, advice, or experiences that the Kafka
> community
> > can share on this topic. Thank you!
> >
> > Regards,
> > Sree Sanjeev.
> >
>
Re: Multiple consumers for a Single Partition in Kafka
Posted by Neeraj Vaidya <ne...@yahoo.co.in.INVALID>.
Hi Sree Sanjeev,
Other than the info from Colt, why can't you just increase partitions and scale the number of consumers accordingly ?
Or build your application as a KStreams application and take advantage of horizontal and vertical scalability ?
Regards,
Neeraj
On Wednesday, 4 October, 2023 at 01:19:37 am GMT+11, Colt McNealy <co...@littlehorse.io> wrote:
Hello,
Currently, you cannot have more than one consumer consuming from a
partition in Kafka. However, there is active discussion about a proposal to
enable queueing semantics for Kafka, which is a superset of the feature you
are talking about. Feel free to check KIP-932:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
Hope this helps,
Colt McNealy
*Founder, LittleHorse.dev*
On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan <
sreesanjeevkg@gmail.com> wrote:
> I'm interested in knowing if we can achieve parallelism by allowing
> multiple consumers to subscribe to a single partition of a topic.
>
> To explore potential solutions, I'm considering the following approach.
>
> 1. Implementing a locking mechanism to control access to the offsets in a
> partition to avoid concurrent processing issues.
>
> My primary concern is maintaining message ordering within the partition, as
> Kafka guarantees ordering within a partition. and message ordering is not
> the primary concern for many applications, so can we have a flag to enable
> multiple consumers to a single partition ?
>
> I appreciate any insights, advice, or experiences that the Kafka community
> can share on this topic. Thank you!
>
> Regards,
> Sree Sanjeev.
>
Re: Multiple consumers for a Single Partition in Kafka
Posted by Colt McNealy <co...@littlehorse.io>.
Hello,
Currently, you cannot have more than one consumer consuming from a
partition in Kafka. However, there is active discussion about a proposal to
enable queueing semantics for Kafka, which is a superset of the feature you
are talking about. Feel free to check KIP-932:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
Hope this helps,
Colt McNealy
*Founder, LittleHorse.dev*
On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan <
sreesanjeevkg@gmail.com> wrote:
> I'm interested in knowing if we can achieve parallelism by allowing
> multiple consumers to subscribe to a single partition of a topic.
>
> To explore potential solutions, I'm considering the following approach.
>
> 1. Implementing a locking mechanism to control access to the offsets in a
> partition to avoid concurrent processing issues.
>
> My primary concern is maintaining message ordering within the partition, as
> Kafka guarantees ordering within a partition. and message ordering is not
> the primary concern for many applications, so can we have a flag to enable
> multiple consumers to a single partition ?
>
> I appreciate any insights, advice, or experiences that the Kafka community
> can share on this topic. Thank you!
>
> Regards,
> Sree Sanjeev.
>
Re: Multiple consumers for a Single Partition in Kafka
Posted by Haruki Okada <oc...@gmail.com>.
Hi Sree.
Yeah, parallel-processing-per-partition requirement often arises
particularly when the throughput is limited due to external I/O latency and
there are some solutions:
- https://github.com/line/decaton
* provides per-key ordering guarantee (or can be unordered if no
ordering is necessary)
* also refs:
https://speakerdeck.com/line_devday2020/decaton-high-performance-task-processing-with-kafka
- https://github.com/confluentinc/parallel-consumer
* also seems to provide per-key ordering guarantee or can be unordered
2023年10月3日(火) 23:11 Sree Sanjeev Kandasamy Gokulrajan <
sreesanjeevkg@gmail.com>:
> I'm interested in knowing if we can achieve parallelism by allowing
> multiple consumers to subscribe to a single partition of a topic.
>
> To explore potential solutions, I'm considering the following approach.
>
> 1. Implementing a locking mechanism to control access to the offsets in a
> partition to avoid concurrent processing issues.
>
> My primary concern is maintaining message ordering within the partition, as
> Kafka guarantees ordering within a partition. and message ordering is not
> the primary concern for many applications, so can we have a flag to enable
> multiple consumers to a single partition ?
>
> I appreciate any insights, advice, or experiences that the Kafka community
> can share on this topic. Thank you!
>
> Regards,
> Sree Sanjeev.
>
--
========================
Okada Haruki
ocadaruma@gmail.com
========================