You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Damian Guy <da...@gmail.com> on 2015/08/25 11:29:25 UTC
0.8.x
Hi,
We currently run 0.7.x on our clusters and are now finally getting around
to upgrading to kafka latest. One thing that has been holding us back is
that we can no longer use a VIP to front the clusters. I understand we
could use a VIP for metadata lookups, but we have 100,000 + producers to at
least one of our clusters.
So, my question is: How is Kafka 0.8.x going to handle 100,000+ producers?
Any recommendations on setup etc?
Thanks for the help,
Damian
Re: 0.8.x
Posted by Damian Guy <da...@gmail.com>.
Hi Ben,
Yes we have a VIP fronting our kafka 0.7 clusters. The producers connect
with a broker list that just points to the VIP.
In 0.8, with the new producer at least, the producer doesn't go through
Zookeeper - which is good. However, all producers will connect to all
brokers to send messages. My concern is that the brokers won't be able to
handle the load of 100k+ connections.
So i want to know if anyone has experience with this sort of load? Will it
work?
Do i need to take a multi-layered approach to Kafka? i.e., have many
front-end kafka clusters that subsets of producers connect to and then
mirror data from there to one or more back-end clusters? I'd rather not
have to do this as it becomes painful to track data though the system.
Any other recommendations on how to go about it?
Thanks,
Damian
On 26 August 2015 at 10:57, Ben Stopford <be...@confluent.io> wrote:
> Hi Damian
>
> Just clarifying - you’re saying you currently have Kafka 0.7.x running
> with a dedicated broker addresses (bypassing ZK) and hitting a VIP which
> you use for load balancing writes. Is that correct?
>
> Are you worried about something specific in the 0.8.x way of doing things
> (ZK under that level of load etc) or are you just looking for experiences
> running 0.8.x with that many producers?
>
> B
>
>
> > On 25 Aug 2015, at 10:29, Damian Guy <da...@gmail.com> wrote:
> >
> > Hi,
> >
> > We currently run 0.7.x on our clusters and are now finally getting around
> > to upgrading to kafka latest. One thing that has been holding us back is
> > that we can no longer use a VIP to front the clusters. I understand we
> > could use a VIP for metadata lookups, but we have 100,000 + producers to
> at
> > least one of our clusters.
> > So, my question is: How is Kafka 0.8.x going to handle 100,000+
> producers?
> > Any recommendations on setup etc?
> >
> > Thanks for the help,
> > Damian
>
>
Re: 0.8.x
Posted by Ben Stopford <be...@confluent.io>.
That’s a fair few connections indeed!
You may be able to take the route of pinning producers to specific partitions. KIP-22 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-22+-+Expose+a+Partitioner+interface+in+the+new+producer> made this easier by exposing a partitioner interface to producers. That should let you balance load.
Also it may be worth adding that the code all uses non-blocking IO. I don’t have hard numbers here though.
Has anyone else worked with 0.8.x at this level of load?
B
> On 26 Aug 2015, at 10:57, Ben Stopford <be...@confluent.io> wrote:
>
> Hi Damian
>
> Just clarifying - you’re saying you currently have Kafka 0.7.x running with a dedicated broker addresses (bypassing ZK) and hitting a VIP which you use for load balancing writes. Is that correct?
>
> Are you worried about something specific in the 0.8.x way of doing things (ZK under that level of load etc) or are you just looking for experiences running 0.8.x with that many producers?
>
> B
>
>
>> On 25 Aug 2015, at 10:29, Damian Guy <da...@gmail.com> wrote:
>>
>> Hi,
>>
>> We currently run 0.7.x on our clusters and are now finally getting around
>> to upgrading to kafka latest. One thing that has been holding us back is
>> that we can no longer use a VIP to front the clusters. I understand we
>> could use a VIP for metadata lookups, but we have 100,000 + producers to at
>> least one of our clusters.
>> So, my question is: How is Kafka 0.8.x going to handle 100,000+ producers?
>> Any recommendations on setup etc?
>>
>> Thanks for the help,
>> Damian
>
Re: 0.8.x
Posted by Ben Stopford <be...@confluent.io>.
Hi Damian
Just clarifying - you’re saying you currently have Kafka 0.7.x running with a dedicated broker addresses (bypassing ZK) and hitting a VIP which you use for load balancing writes. Is that correct?
Are you worried about something specific in the 0.8.x way of doing things (ZK under that level of load etc) or are you just looking for experiences running 0.8.x with that many producers?
B
> On 25 Aug 2015, at 10:29, Damian Guy <da...@gmail.com> wrote:
>
> Hi,
>
> We currently run 0.7.x on our clusters and are now finally getting around
> to upgrading to kafka latest. One thing that has been holding us back is
> that we can no longer use a VIP to front the clusters. I understand we
> could use a VIP for metadata lookups, but we have 100,000 + producers to at
> least one of our clusters.
> So, my question is: How is Kafka 0.8.x going to handle 100,000+ producers?
> Any recommendations on setup etc?
>
> Thanks for the help,
> Damian