Is fetching from in-sync replicas possible?

[Theo Hultberg <th...@iconara.net>]

Hello,

Apologies if this question has been asked before. If I understand things
correctly a client can only fetch from the leader of a partition, not from
an (in-sync) replica. I have a use case where it would be very beneficial
if it were possible to fetch from a replica instead of just the leader, and
I wonder why it is not allowed? Are there any consistency problems with
allowing it, for example? Is there any way to configure Kafka to allow it?

The use case is a Kafka cluster running in EC2 across three availability
zones. I would like clients to only fetch from a replica in the same zone
to avoid paying for more inter-zone traffic than absolutely necessary. The
producer side and the replication will still send traffic across zones, but
it would be nice to be able to avoid another zone crossing on the consumer
side.

We currently have a Kafka cluster that costs about the same in bandwidth
cost as running the instances, so every little bit helps.

yours,
Theo

Re: Is fetching from in-sync replicas possible?

[James Cheng <jc...@tivo.com>]

On May 27, 2015, at 11:23 AM, Joel Koshy <jj...@gmail.com> wrote:

> That's right - it should not help significantly assuming even
> distribution of leaders and even distribution of partition volume
> (average inbound messages/sec).
> 

Aditya, Joel,

Oh, right, that makes sense. If I had a 10 partition topic across 10 nodes where each leader handles 1/10th of the consumer traffic for that topic, I could change that and instead have 100 partition topic across 100 nodes, and then each leader would only have to handle 1/100th of the consumer traffic for that topic.

-James


> Theo's use-case is a bit different though in which you want to avoid
> cross-zone consumer reads especially if you have a high fan-out in
> number of consumers.
> 
> On Wed, May 27, 2015 at 05:56:56PM +0000, Aditya Auradkar wrote:
>> Is that necessarily the case? On a cluster hosting partitions, assuming the leaders are evenly distributed, every node should receive a roughly equal share of the traffic. It does help a lot when the consumer throughput of a single partition exceeds the capacity of a single leader but at that point the topic ideally needs more partitions.
>> 
>> Aditya
>> 
>> ________________________________________
>> From: James Cheng [jcheng@tivo.com]
>> Sent: Wednesday, May 27, 2015 10:50 AM
>> To: users@kafka.apache.org
>> Subject: Re: Is fetching from in-sync replicas possible?
>> 
>> On May 26, 2015, at 1:44 PM, Joel Koshy <jj...@gmail.com> wrote:
>> 
>>>> Apologies if this question has been asked before. If I understand things
>>>> correctly a client can only fetch from the leader of a partition, not from
>>>> an (in-sync) replica. I have a use case where it would be very beneficial
>>>> if it were possible to fetch from a replica instead of just the leader, and
>>>> I wonder why it is not allowed? Are there any consistency problems with
>>>> allowing it, for example? Is there any way to configure Kafka to allow it?
>>> 
>>> Yes this should be possible.  I don't think there are any consistency
>>> issues (barring any bugs) since we never expose past the
>>> high-watermark and the follower HW is strictly <= leader HW. Can you
>>> file a jira for this?
>>> 
>> 
>> Wouldn't this allow Kafka to scale to handle a lot more consumer traffic? Currently, consumers all have to read from the leader, which means that the network/disk bandwidth of a particular leader is the bottleneck. If consumers could read from in-sync replicas, then a single node no longer is the bottleneck for reads. You could scale out your read capacity as far as you want.
>> 
>> -James
>> 
>> 
>>>> The use case is a Kafka cluster running in EC2 across three availability
>>>> zones.
>>> 
>>> Out of curiosity - what's the typical latency (distribution) you see
>>> between zones?
>>> 
>>> Joel
>> 
> 

Re: Is fetching from in-sync replicas possible?

[Joel Koshy <jj...@gmail.com>]

That's right - it should not help significantly assuming even
distribution of leaders and even distribution of partition volume
(average inbound messages/sec).

Theo's use-case is a bit different though in which you want to avoid
cross-zone consumer reads especially if you have a high fan-out in
number of consumers.

On Wed, May 27, 2015 at 05:56:56PM +0000, Aditya Auradkar wrote:
> Is that necessarily the case? On a cluster hosting partitions, assuming the leaders are evenly distributed, every node should receive a roughly equal share of the traffic. It does help a lot when the consumer throughput of a single partition exceeds the capacity of a single leader but at that point the topic ideally needs more partitions.
> 
> Aditya
> 
> ________________________________________
> From: James Cheng [jcheng@tivo.com]
> Sent: Wednesday, May 27, 2015 10:50 AM
> To: users@kafka.apache.org
> Subject: Re: Is fetching from in-sync replicas possible?
> 
> On May 26, 2015, at 1:44 PM, Joel Koshy <jj...@gmail.com> wrote:
> 
> >> Apologies if this question has been asked before. If I understand things
> >> correctly a client can only fetch from the leader of a partition, not from
> >> an (in-sync) replica. I have a use case where it would be very beneficial
> >> if it were possible to fetch from a replica instead of just the leader, and
> >> I wonder why it is not allowed? Are there any consistency problems with
> >> allowing it, for example? Is there any way to configure Kafka to allow it?
> >
> > Yes this should be possible.  I don't think there are any consistency
> > issues (barring any bugs) since we never expose past the
> > high-watermark and the follower HW is strictly <= leader HW. Can you
> > file a jira for this?
> >
> 
> Wouldn't this allow Kafka to scale to handle a lot more consumer traffic? Currently, consumers all have to read from the leader, which means that the network/disk bandwidth of a particular leader is the bottleneck. If consumers could read from in-sync replicas, then a single node no longer is the bottleneck for reads. You could scale out your read capacity as far as you want.
> 
> -James
> 
> 
> >> The use case is a Kafka cluster running in EC2 across three availability
> >> zones.
> >
> > Out of curiosity - what's the typical latency (distribution) you see
> > between zones?
> >
> > Joel
> 

RE: Is fetching from in-sync replicas possible?

[Aditya Auradkar <aa...@linkedin.com.INVALID>]

Is that necessarily the case? On a cluster hosting partitions, assuming the leaders are evenly distributed, every node should receive a roughly equal share of the traffic. It does help a lot when the consumer throughput of a single partition exceeds the capacity of a single leader but at that point the topic ideally needs more partitions.

Aditya

________________________________________
From: James Cheng [jcheng@tivo.com]
Sent: Wednesday, May 27, 2015 10:50 AM
To: users@kafka.apache.org
Subject: Re: Is fetching from in-sync replicas possible?

On May 26, 2015, at 1:44 PM, Joel Koshy <jj...@gmail.com> wrote:

>> Apologies if this question has been asked before. If I understand things
>> correctly a client can only fetch from the leader of a partition, not from
>> an (in-sync) replica. I have a use case where it would be very beneficial
>> if it were possible to fetch from a replica instead of just the leader, and
>> I wonder why it is not allowed? Are there any consistency problems with
>> allowing it, for example? Is there any way to configure Kafka to allow it?
>
> Yes this should be possible.  I don't think there are any consistency
> issues (barring any bugs) since we never expose past the
> high-watermark and the follower HW is strictly <= leader HW. Can you
> file a jira for this?
>

Wouldn't this allow Kafka to scale to handle a lot more consumer traffic? Currently, consumers all have to read from the leader, which means that the network/disk bandwidth of a particular leader is the bottleneck. If consumers could read from in-sync replicas, then a single node no longer is the bottleneck for reads. You could scale out your read capacity as far as you want.

-James


>> The use case is a Kafka cluster running in EC2 across three availability
>> zones.
>
> Out of curiosity - what's the typical latency (distribution) you see
> between zones?
>
> Joel

Re: Is fetching from in-sync replicas possible?

[James Cheng <jc...@tivo.com>]

On May 26, 2015, at 1:44 PM, Joel Koshy <jj...@gmail.com> wrote:

>> Apologies if this question has been asked before. If I understand things
>> correctly a client can only fetch from the leader of a partition, not from
>> an (in-sync) replica. I have a use case where it would be very beneficial
>> if it were possible to fetch from a replica instead of just the leader, and
>> I wonder why it is not allowed? Are there any consistency problems with
>> allowing it, for example? Is there any way to configure Kafka to allow it?
> 
> Yes this should be possible.  I don't think there are any consistency
> issues (barring any bugs) since we never expose past the
> high-watermark and the follower HW is strictly <= leader HW. Can you
> file a jira for this?
> 

Wouldn't this allow Kafka to scale to handle a lot more consumer traffic? Currently, consumers all have to read from the leader, which means that the network/disk bandwidth of a particular leader is the bottleneck. If consumers could read from in-sync replicas, then a single node no longer is the bottleneck for reads. You could scale out your read capacity as far as you want.

-James


>> The use case is a Kafka cluster running in EC2 across three availability
>> zones.
> 
> Out of curiosity - what's the typical latency (distribution) you see
> between zones?
> 
> Joel

Re: Is fetching from in-sync replicas possible?

[Joel Koshy <jj...@gmail.com>]

> > Out of curiosity - what's the typical latency (distribution) you see
> between zones?
> 
> Unfortunately I don't have any good numbers on that. Since we're publishing
> both in the same AZ and to other AZs the latency metrics reflect both. If I
> figure out a good way to measure this I will report back.

Thanks - I was wondering if you had done some simple ping/traceroute
tests. It's just that with replicas in different zones the end-to-end
latency from producer to consumer will be correspondingly higher.
Does your zookeeper setup span zones as well?

> 
> On Tue, May 26, 2015 at 10:44 PM, Joel Koshy <jj...@gmail.com> wrote:
> 
> > > Apologies if this question has been asked before. If I understand things
> > > correctly a client can only fetch from the leader of a partition, not
> > from
> > > an (in-sync) replica. I have a use case where it would be very beneficial
> > > if it were possible to fetch from a replica instead of just the leader,
> > and
> > > I wonder why it is not allowed? Are there any consistency problems with
> > > allowing it, for example? Is there any way to configure Kafka to allow
> > it?
> >
> > Yes this should be possible.  I don't think there are any consistency
> > issues (barring any bugs) since we never expose past the
> > high-watermark and the follower HW is strictly <= leader HW. Can you
> > file a jira for this?
> >
> > > The use case is a Kafka cluster running in EC2 across three availability
> > > zones.
> >
> > Out of curiosity - what's the typical latency (distribution) you see
> > between zones?
> >
> > Joel
> >

Re: Is fetching from in-sync replicas possible?

[Theo Hultberg <th...@iconara.net>]

Thank you for your response Joel,

> Can you file a jira for this?

I've created https://issues.apache.org/jira/browse/KAFKA-2225

> Out of curiosity - what's the typical latency (distribution) you see
between zones?

Unfortunately I don't have any good numbers on that. Since we're publishing
both in the same AZ and to other AZs the latency metrics reflect both. If I
figure out a good way to measure this I will report back.

T#

On Tue, May 26, 2015 at 10:44 PM, Joel Koshy <jj...@gmail.com> wrote:

> > Apologies if this question has been asked before. If I understand things
> > correctly a client can only fetch from the leader of a partition, not
> from
> > an (in-sync) replica. I have a use case where it would be very beneficial
> > if it were possible to fetch from a replica instead of just the leader,
> and
> > I wonder why it is not allowed? Are there any consistency problems with
> > allowing it, for example? Is there any way to configure Kafka to allow
> it?
>
> Yes this should be possible.  I don't think there are any consistency
> issues (barring any bugs) since we never expose past the
> high-watermark and the follower HW is strictly <= leader HW. Can you
> file a jira for this?
>
> > The use case is a Kafka cluster running in EC2 across three availability
> > zones.
>
> Out of curiosity - what's the typical latency (distribution) you see
> between zones?
>
> Joel
>

Re: Is fetching from in-sync replicas possible?

[Joel Koshy <jj...@gmail.com>]

> Apologies if this question has been asked before. If I understand things
> correctly a client can only fetch from the leader of a partition, not from
> an (in-sync) replica. I have a use case where it would be very beneficial
> if it were possible to fetch from a replica instead of just the leader, and
> I wonder why it is not allowed? Are there any consistency problems with
> allowing it, for example? Is there any way to configure Kafka to allow it?

Yes this should be possible.  I don't think there are any consistency
issues (barring any bugs) since we never expose past the
high-watermark and the follower HW is strictly <= leader HW. Can you
file a jira for this?

> The use case is a Kafka cluster running in EC2 across three availability
> zones.

Out of curiosity - what's the typical latency (distribution) you see
between zones?

Joel