You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Mohit Chawla <mo...@gmail.com> on 2017/08/22 10:14:08 UTC
Pinning clients to specific brokers
Hi folks,
Is it possible to pin kafka clients to use only specific brokers throughout
their lifetime and not just for the initial bootstrapping ?
Thanks,
Mohit
Re: Pinning clients to specific brokers
Posted by Mohit Chawla <mo...@gmail.com>.
Hi Stephen, Hans,
This can help in keeping the traffic isolated in an availability zone, and
avoid inter AZ communication.
Thanks,
Mohit
On Thursday, August 24, 2017, Hans Jespersen <ha...@confluent.io> wrote:
> We (Confluent) run Kafka as a SaaS-based cloud offering and we do not see
> any reason for this feature so I just don’t understand the motivation for
> it. Please explain.
>
> -hans
>
> --
> /**
> * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
> * hans@confluent.io <javascript:;> (650)924-2670
> */
>
>
>
>
> > On Aug 23, 2017, at 12:42 AM, Mohit Chawla <
> mohit.chawla.binary@gmail.com <javascript:;>> wrote:
> >
> > Hey Hans,
> >
> > Thanks for your reply.
> >
> > In a cloud environment this can be useful. Perhaps if partitioning and
> > replicas are selected appropriately, this could be possible ?
> >
> > Thanks,
> > Mohit
> >
> > On Tuesday, August 22, 2017, Hans Jespersen <hans@confluent.io
> <javascript:;>> wrote:
> >
> >> Doing that doesn't really make sense in a Kafka cluster because the
> topic
> >> partitions and their replicas are spread out across many brokers in the
> >> cluster. That's what enables the parallel processing and fault tolerance
> >> features of Kafka.
> >>
> >> -hans
> >>
> >>> On Aug 22, 2017, at 3:14 AM, Mohit Chawla <
> mohit.chawla.binary@gmail.com <javascript:;>
> >> <javascript:;>> wrote:
> >>>
> >>> Hi folks,
> >>>
> >>> Is it possible to pin kafka clients to use only specific brokers
> >> throughout
> >>> their lifetime and not just for the initial bootstrapping ?
> >>>
> >>> Thanks,
> >>> Mohit
> >>
>
>
Re: Pinning clients to specific brokers
Posted by Hans Jespersen <ha...@confluent.io>.
We (Confluent) run Kafka as a SaaS-based cloud offering and we do not see any reason for this feature so I just don’t understand the motivation for it. Please explain.
-hans
--
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* hans@confluent.io (650)924-2670
*/
> On Aug 23, 2017, at 12:42 AM, Mohit Chawla <mo...@gmail.com> wrote:
>
> Hey Hans,
>
> Thanks for your reply.
>
> In a cloud environment this can be useful. Perhaps if partitioning and
> replicas are selected appropriately, this could be possible ?
>
> Thanks,
> Mohit
>
> On Tuesday, August 22, 2017, Hans Jespersen <ha...@confluent.io> wrote:
>
>> Doing that doesn't really make sense in a Kafka cluster because the topic
>> partitions and their replicas are spread out across many brokers in the
>> cluster. That's what enables the parallel processing and fault tolerance
>> features of Kafka.
>>
>> -hans
>>
>>> On Aug 22, 2017, at 3:14 AM, Mohit Chawla <mohit.chawla.binary@gmail.com
>> <javascript:;>> wrote:
>>>
>>> Hi folks,
>>>
>>> Is it possible to pin kafka clients to use only specific brokers
>> throughout
>>> their lifetime and not just for the initial bootstrapping ?
>>>
>>> Thanks,
>>> Mohit
>>
Re: Pinning clients to specific brokers
Posted by Stephen Durfey <sj...@gmail.com>.
Mohit,
Can you describe your use case around why you want this to happen?
Thanks
________________________________
From: Joao Reis <jo...@blip.pt>
Sent: Wednesday, August 23, 2017 11:08:02 AM
To: users@kafka.apache.org
Subject: Re: Pinning clients to specific brokers
Hey Mohit,
I agree with Hans, and additionally you may have trouble when the cluster needs to rebalance its partitions or when partition leadership changes node.
Also you may loose high-availability and cluster resilience in case of fail-over.
Cheers
João Reis
On 23 Aug 2017, at 08:42, Mohit Chawla <mo...@gmail.com>> wrote:
Hey Hans,
Thanks for your reply.
In a cloud environment this can be useful. Perhaps if partitioning and
replicas are selected appropriately, this could be possible ?
Thanks,
Mohit
On Tuesday, August 22, 2017, Hans Jespersen <ha...@confluent.io>> wrote:
Doing that doesn't really make sense in a Kafka cluster because the topic
partitions and their replicas are spread out across many brokers in the
cluster. That's what enables the parallel processing and fault tolerance
features of Kafka.
-hans
On Aug 22, 2017, at 3:14 AM, Mohit Chawla <mo...@gmail.com>
<javascript:;>> wrote:
Hi folks,
Is it possible to pin kafka clients to use only specific brokers
throughout
their lifetime and not just for the initial bootstrapping ?
Thanks,
Mohit
______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com<http://www.symanteccloud.com/>
______________________________________________________________________
______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________
Re: Pinning clients to specific brokers
Posted by Joao Reis <jo...@blip.pt>.
Hey Mohit,
I agree with Hans, and additionally you may have trouble when the cluster needs to rebalance its partitions or when partition leadership changes node.
Also you may loose high-availability and cluster resilience in case of fail-over.
Cheers
João Reis
On 23 Aug 2017, at 08:42, Mohit Chawla <mo...@gmail.com>> wrote:
Hey Hans,
Thanks for your reply.
In a cloud environment this can be useful. Perhaps if partitioning and
replicas are selected appropriately, this could be possible ?
Thanks,
Mohit
On Tuesday, August 22, 2017, Hans Jespersen <ha...@confluent.io>> wrote:
Doing that doesn't really make sense in a Kafka cluster because the topic
partitions and their replicas are spread out across many brokers in the
cluster. That's what enables the parallel processing and fault tolerance
features of Kafka.
-hans
On Aug 22, 2017, at 3:14 AM, Mohit Chawla <mo...@gmail.com>
<javascript:;>> wrote:
Hi folks,
Is it possible to pin kafka clients to use only specific brokers
throughout
their lifetime and not just for the initial bootstrapping ?
Thanks,
Mohit
______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com<http://www.symanteccloud.com/>
______________________________________________________________________
______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________
Re: Pinning clients to specific brokers
Posted by Mohit Chawla <mo...@gmail.com>.
Hey Hans,
Thanks for your reply.
In a cloud environment this can be useful. Perhaps if partitioning and
replicas are selected appropriately, this could be possible ?
Thanks,
Mohit
On Tuesday, August 22, 2017, Hans Jespersen <ha...@confluent.io> wrote:
> Doing that doesn't really make sense in a Kafka cluster because the topic
> partitions and their replicas are spread out across many brokers in the
> cluster. That's what enables the parallel processing and fault tolerance
> features of Kafka.
>
> -hans
>
> > On Aug 22, 2017, at 3:14 AM, Mohit Chawla <mohit.chawla.binary@gmail.com
> <javascript:;>> wrote:
> >
> > Hi folks,
> >
> > Is it possible to pin kafka clients to use only specific brokers
> throughout
> > their lifetime and not just for the initial bootstrapping ?
> >
> > Thanks,
> > Mohit
>
Re: Pinning clients to specific brokers
Posted by Hans Jespersen <ha...@confluent.io>.
Doing that doesn't really make sense in a Kafka cluster because the topic partitions and their replicas are spread out across many brokers in the cluster. That's what enables the parallel processing and fault tolerance features of Kafka.
-hans
> On Aug 22, 2017, at 3:14 AM, Mohit Chawla <mo...@gmail.com> wrote:
>
> Hi folks,
>
> Is it possible to pin kafka clients to use only specific brokers throughout
> their lifetime and not just for the initial bootstrapping ?
>
> Thanks,
> Mohit