You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Tony John <to...@gmail.com> on 2020/10/19 16:20:28 UTC
Kafka Streams : Rack Aware deployment in EC2
Hi All,
I have been trying to figure out some documentation on how to enable rack
awareness on a Kafka Streams app. I do see the broker.rack configuration
which needs to be done at the broker side and client.rack configuration for
the consumers. Is there any specific configuration which is required to
enable rack awareness in a streams app? Please throw some light on this.
PS: I am using Kafka / Kafka Streams 2.2.1
Thanks,
Tony
Re: Kafka Streams : Rack Aware deployment in EC2
Posted by Tony John <to...@gmail.com>.
Thanks a lot Guozhang. Will give it a try.
Thanks,
Tony
On Tue, Oct 20, 2020 at 10:42 PM Guozhang Wang <wa...@gmail.com> wrote:
> Hello Tony,
>
> I think you already know the consumer-client side fetch-from-follower
> feature:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
>
> Say your Kafka deployment is cross-AZs, and your Streams is also deployed
> cross-AZ, you can, by setting the "client.rack" through streams configs, to
> let the embedded consumers to read from close replicas within the same AZ
> as well.
>
> Guozhang
>
>
> On Mon, Oct 19, 2020 at 9:19 PM Tony John <to...@gmail.com>
> wrote:
>
> > Thanks Matthias.. I don't think I will be able to take it up.. Will wait
> > for it to be available in future. :)
> >
> > On Mon, Oct 19, 2020 at 11:56 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> > > No suggestions... Sorry.
> > >
> > > The task assignment algorithm cannot be customized.
> > >
> > > The only thing you _could_ do, is to pickup the ticket yourself (to get
> > > the feature maybe in 2.8 release). Not sure if you would be interested
> > > to contribute :)
> > >
> > >
> > > -Matthias
> > >
> > > On 10/19/20 11:08 AM, Tony John wrote:
> > > > Thanks for the quick response Matthias. We were planning to move to
> AWS
> > > > MSK, and I think we will have to defer it until rack awareness is in
> > > place
> > > > for Kafka Streams. Reason being consumers consuming from brokers in
> > > > different availability zones would result in high data transfer
> > charges.
> > > Do
> > > > you have any suggestions?
> > > >
> > > > Thanks,
> > > > Tony
> > > >
> > > > On Mon, Oct 19, 2020 at 11:08 PM Matthias J. Sax <mj...@apache.org>
> > > wrote:
> > > >
> > > >> There is no such feature for Kafka Streams yet.
> > > >>
> > > >> We have one ticket for rack aware standby task placement
> > > >> (https://issues.apache.org/jira/browse/KAFKA-6718) but nobody is
> > > working
> > > >> on it.
> > > >>
> > > >>
> > > >> -Matthias
> > > >>
> > > >> On 10/19/20 9:20 AM, Tony John wrote:
> > > >>> Hi All,
> > > >>>
> > > >>> I have been trying to figure out some documentation on how to
> enable
> > > rack
> > > >>> awareness on a Kafka Streams app. I do see the broker.rack
> > > configuration
> > > >>> which needs to be done at the broker side and client.rack
> > configuration
> > > >> for
> > > >>> the consumers. Is there any specific configuration which is
> required
> > to
> > > >>> enable rack awareness in a streams app? Please throw some light on
> > > this.
> > > >>>
> > > >>> PS: I am using Kafka / Kafka Streams 2.2.1
> > > >>>
> > > >>> Thanks,
> > > >>> Tony
> > > >>>
> > > >>
> > > >
> > >
> >
>
>
> --
> -- Guozhang
>
Re: Kafka Streams : Rack Aware deployment in EC2
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Tony,
I think you already know the consumer-client side fetch-from-follower
feature:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
Say your Kafka deployment is cross-AZs, and your Streams is also deployed
cross-AZ, you can, by setting the "client.rack" through streams configs, to
let the embedded consumers to read from close replicas within the same AZ
as well.
Guozhang
On Mon, Oct 19, 2020 at 9:19 PM Tony John <to...@gmail.com> wrote:
> Thanks Matthias.. I don't think I will be able to take it up.. Will wait
> for it to be available in future. :)
>
> On Mon, Oct 19, 2020 at 11:56 PM Matthias J. Sax <mj...@apache.org> wrote:
>
> > No suggestions... Sorry.
> >
> > The task assignment algorithm cannot be customized.
> >
> > The only thing you _could_ do, is to pickup the ticket yourself (to get
> > the feature maybe in 2.8 release). Not sure if you would be interested
> > to contribute :)
> >
> >
> > -Matthias
> >
> > On 10/19/20 11:08 AM, Tony John wrote:
> > > Thanks for the quick response Matthias. We were planning to move to AWS
> > > MSK, and I think we will have to defer it until rack awareness is in
> > place
> > > for Kafka Streams. Reason being consumers consuming from brokers in
> > > different availability zones would result in high data transfer
> charges.
> > Do
> > > you have any suggestions?
> > >
> > > Thanks,
> > > Tony
> > >
> > > On Mon, Oct 19, 2020 at 11:08 PM Matthias J. Sax <mj...@apache.org>
> > wrote:
> > >
> > >> There is no such feature for Kafka Streams yet.
> > >>
> > >> We have one ticket for rack aware standby task placement
> > >> (https://issues.apache.org/jira/browse/KAFKA-6718) but nobody is
> > working
> > >> on it.
> > >>
> > >>
> > >> -Matthias
> > >>
> > >> On 10/19/20 9:20 AM, Tony John wrote:
> > >>> Hi All,
> > >>>
> > >>> I have been trying to figure out some documentation on how to enable
> > rack
> > >>> awareness on a Kafka Streams app. I do see the broker.rack
> > configuration
> > >>> which needs to be done at the broker side and client.rack
> configuration
> > >> for
> > >>> the consumers. Is there any specific configuration which is required
> to
> > >>> enable rack awareness in a streams app? Please throw some light on
> > this.
> > >>>
> > >>> PS: I am using Kafka / Kafka Streams 2.2.1
> > >>>
> > >>> Thanks,
> > >>> Tony
> > >>>
> > >>
> > >
> >
>
--
-- Guozhang
Re: Kafka Streams : Rack Aware deployment in EC2
Posted by Tony John <to...@gmail.com>.
Thanks Matthias.. I don't think I will be able to take it up.. Will wait
for it to be available in future. :)
On Mon, Oct 19, 2020 at 11:56 PM Matthias J. Sax <mj...@apache.org> wrote:
> No suggestions... Sorry.
>
> The task assignment algorithm cannot be customized.
>
> The only thing you _could_ do, is to pickup the ticket yourself (to get
> the feature maybe in 2.8 release). Not sure if you would be interested
> to contribute :)
>
>
> -Matthias
>
> On 10/19/20 11:08 AM, Tony John wrote:
> > Thanks for the quick response Matthias. We were planning to move to AWS
> > MSK, and I think we will have to defer it until rack awareness is in
> place
> > for Kafka Streams. Reason being consumers consuming from brokers in
> > different availability zones would result in high data transfer charges.
> Do
> > you have any suggestions?
> >
> > Thanks,
> > Tony
> >
> > On Mon, Oct 19, 2020 at 11:08 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> >> There is no such feature for Kafka Streams yet.
> >>
> >> We have one ticket for rack aware standby task placement
> >> (https://issues.apache.org/jira/browse/KAFKA-6718) but nobody is
> working
> >> on it.
> >>
> >>
> >> -Matthias
> >>
> >> On 10/19/20 9:20 AM, Tony John wrote:
> >>> Hi All,
> >>>
> >>> I have been trying to figure out some documentation on how to enable
> rack
> >>> awareness on a Kafka Streams app. I do see the broker.rack
> configuration
> >>> which needs to be done at the broker side and client.rack configuration
> >> for
> >>> the consumers. Is there any specific configuration which is required to
> >>> enable rack awareness in a streams app? Please throw some light on
> this.
> >>>
> >>> PS: I am using Kafka / Kafka Streams 2.2.1
> >>>
> >>> Thanks,
> >>> Tony
> >>>
> >>
> >
>
Re: Kafka Streams : Rack Aware deployment in EC2
Posted by "Matthias J. Sax" <mj...@apache.org>.
No suggestions... Sorry.
The task assignment algorithm cannot be customized.
The only thing you _could_ do, is to pickup the ticket yourself (to get
the feature maybe in 2.8 release). Not sure if you would be interested
to contribute :)
-Matthias
On 10/19/20 11:08 AM, Tony John wrote:
> Thanks for the quick response Matthias. We were planning to move to AWS
> MSK, and I think we will have to defer it until rack awareness is in place
> for Kafka Streams. Reason being consumers consuming from brokers in
> different availability zones would result in high data transfer charges. Do
> you have any suggestions?
>
> Thanks,
> Tony
>
> On Mon, Oct 19, 2020 at 11:08 PM Matthias J. Sax <mj...@apache.org> wrote:
>
>> There is no such feature for Kafka Streams yet.
>>
>> We have one ticket for rack aware standby task placement
>> (https://issues.apache.org/jira/browse/KAFKA-6718) but nobody is working
>> on it.
>>
>>
>> -Matthias
>>
>> On 10/19/20 9:20 AM, Tony John wrote:
>>> Hi All,
>>>
>>> I have been trying to figure out some documentation on how to enable rack
>>> awareness on a Kafka Streams app. I do see the broker.rack configuration
>>> which needs to be done at the broker side and client.rack configuration
>> for
>>> the consumers. Is there any specific configuration which is required to
>>> enable rack awareness in a streams app? Please throw some light on this.
>>>
>>> PS: I am using Kafka / Kafka Streams 2.2.1
>>>
>>> Thanks,
>>> Tony
>>>
>>
>
Re: Kafka Streams : Rack Aware deployment in EC2
Posted by Tony John <to...@gmail.com>.
Thanks for the quick response Matthias. We were planning to move to AWS
MSK, and I think we will have to defer it until rack awareness is in place
for Kafka Streams. Reason being consumers consuming from brokers in
different availability zones would result in high data transfer charges. Do
you have any suggestions?
Thanks,
Tony
On Mon, Oct 19, 2020 at 11:08 PM Matthias J. Sax <mj...@apache.org> wrote:
> There is no such feature for Kafka Streams yet.
>
> We have one ticket for rack aware standby task placement
> (https://issues.apache.org/jira/browse/KAFKA-6718) but nobody is working
> on it.
>
>
> -Matthias
>
> On 10/19/20 9:20 AM, Tony John wrote:
> > Hi All,
> >
> > I have been trying to figure out some documentation on how to enable rack
> > awareness on a Kafka Streams app. I do see the broker.rack configuration
> > which needs to be done at the broker side and client.rack configuration
> for
> > the consumers. Is there any specific configuration which is required to
> > enable rack awareness in a streams app? Please throw some light on this.
> >
> > PS: I am using Kafka / Kafka Streams 2.2.1
> >
> > Thanks,
> > Tony
> >
>
Re: Kafka Streams : Rack Aware deployment in EC2
Posted by "Matthias J. Sax" <mj...@apache.org>.
There is no such feature for Kafka Streams yet.
We have one ticket for rack aware standby task placement
(https://issues.apache.org/jira/browse/KAFKA-6718) but nobody is working
on it.
-Matthias
On 10/19/20 9:20 AM, Tony John wrote:
> Hi All,
>
> I have been trying to figure out some documentation on how to enable rack
> awareness on a Kafka Streams app. I do see the broker.rack configuration
> which needs to be done at the broker side and client.rack configuration for
> the consumers. Is there any specific configuration which is required to
> enable rack awareness in a streams app? Please throw some light on this.
>
> PS: I am using Kafka / Kafka Streams 2.2.1
>
> Thanks,
> Tony
>