Re: Asking about how to consume the message from follower partition

[Ryanne Dolan <ry...@gmail.com>]

I'm not sure whether this is what the OP is asking for, but some
partitioned systems let you select a specific follower to consume from
_instead_ of from the leader, e.g. couchbase, which can be useful in
scenarios where there are a ton more consumers than producers for a
partition, and you don't want consumers impacting the leader's ability to
accept new events.

It's also not uncommon to have different replicas in different availability
zones or racks. Consumers may prefer reading from their local AZ, even if
that includes followers.

Ryanne

On Sun, Sep 30, 2018, 1:13 PM Matthias J. Sax <ma...@confluent.io> wrote:

> Kafka scales via partitions, not via replication. Replication guarantees
> fault-tolerance. Thus, it's not possible atm, to consume from follower
> partitions -- also note, even if different consumer would read from
> follower partitions, each of them would read the same data. I assume,
> you want different consumers (with a consumer group) to read different
> data. For this, your topic should have multiple partitions.
>
> -Matthias
>
> On 9/30/18 9:08 AM, M. Manna wrote:
> > Have you set the bootstrap.servers to your local broker and check?
> >
> > Also, don’t forget that for any topic there’s a leader and follower - so
> > asking the follower to do a leader’s job isn’t probably the correct
> > intention.
> >
> >
> >
> > On Sun, 30 Sep 2018 at 17:06, Aung Myo Htut <ag...@gmail.com>
> > wrote:
> >
> >> Dear sir
> >>
> >> I'm doing the project of streaming the image sequence using the Apache
> >> Kafka.
> >> Now I can stream with the single broker. To have the scalability, I
> want to
> >> use 3 Kafka Brokers with 3 replication and Brokers will be installed at
> >> different locations. So I want consumers to consume from the local
> broker.
> >> But when I check consumer consume only from the leader partition of the
> >> topic. So how can local consumer pull the messages from the follower
> >> partition?
> >>
> >> Best regards,
> >> Aung Myo Htut
> >>
> >
>
>

Re: Asking about how to consume the message from follower partition

["Matthias J. Sax" <ma...@confluent.io>]

It's a valid feature request. I was just describing the current
implementation that does not support reading from followers.


-Matthias

On 10/1/18 9:06 AM, Ryanne Dolan wrote:
> I'm not sure whether this is what the OP is asking for, but some
> partitioned systems let you select a specific follower to consume from
> _instead_ of from the leader, e.g. couchbase, which can be useful in
> scenarios where there are a ton more consumers than producers for a
> partition, and you don't want consumers impacting the leader's ability to
> accept new events.
> 
> It's also not uncommon to have different replicas in different availability
> zones or racks. Consumers may prefer reading from their local AZ, even if
> that includes followers.
> 
> Ryanne
> 
> On Sun, Sep 30, 2018, 1:13 PM Matthias J. Sax <ma...@confluent.io> wrote:
> 
>> Kafka scales via partitions, not via replication. Replication guarantees
>> fault-tolerance. Thus, it's not possible atm, to consume from follower
>> partitions -- also note, even if different consumer would read from
>> follower partitions, each of them would read the same data. I assume,
>> you want different consumers (with a consumer group) to read different
>> data. For this, your topic should have multiple partitions.
>>
>> -Matthias
>>
>> On 9/30/18 9:08 AM, M. Manna wrote:
>>> Have you set the bootstrap.servers to your local broker and check?
>>>
>>> Also, don’t forget that for any topic there’s a leader and follower - so
>>> asking the follower to do a leader’s job isn’t probably the correct
>>> intention.
>>>
>>>
>>>
>>> On Sun, 30 Sep 2018 at 17:06, Aung Myo Htut <ag...@gmail.com>
>>> wrote:
>>>
>>>> Dear sir
>>>>
>>>> I'm doing the project of streaming the image sequence using the Apache
>>>> Kafka.
>>>> Now I can stream with the single broker. To have the scalability, I
>> want to
>>>> use 3 Kafka Brokers with 3 replication and Brokers will be installed at
>>>> different locations. So I want consumers to consume from the local
>> broker.
>>>> But when I check consumer consume only from the leader partition of the
>>>> topic. So how can local consumer pull the messages from the follower
>>>> partition?
>>>>
>>>> Best regards,
>>>> Aung Myo Htut
>>>>
>>>
>>
>>
>