You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ersin Er <er...@gmail.com> on 2014/07/07 12:33:24 UTC
Largest known Kafka deployment?
Hi,
LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are larger
deployments than LinkedIn's. What's are the largest Kafka deployments you
know of? Any public performance and scalability data published for such
clusters?
Any pointers would be interesting and helpful.
Regards,
--
Ersin Er
Re: Largest known Kafka deployment?
Posted by Ersin Er <er...@gmail.com>.
I mean brokers particularly but others are also welcome.
On Jul 7, 2014 3:36 PM, "Otis Gospodnetic" <ot...@gmail.com>
wrote:
> Hi,
>
> I think it depends on what you mean by largest? Most brokers? Producers?
> Consumers? Messages? Bytes?
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Mon, Jul 7, 2014 at 6:33 AM, Ersin Er <er...@gmail.com> wrote:
>
> > Hi,
> >
> > LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are larger
> > deployments than LinkedIn's. What's are the largest Kafka deployments you
> > know of? Any public performance and scalability data published for such
> > clusters?
> >
> > Any pointers would be interesting and helpful.
> >
> > Regards,
> >
> > --
> > Ersin Er
> >
>
Re: Largest known Kafka deployment?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
I think it depends on what you mean by largest? Most brokers? Producers?
Consumers? Messages? Bytes?
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Jul 7, 2014 at 6:33 AM, Ersin Er <er...@gmail.com> wrote:
> Hi,
>
> LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are larger
> deployments than LinkedIn's. What's are the largest Kafka deployments you
> know of? Any public performance and scalability data published for such
> clusters?
>
> Any pointers would be interesting and helpful.
>
> Regards,
>
> --
> Ersin Er
>
Re: Largest known Kafka deployment?
Posted by Todd Palino <tp...@linkedin.com.INVALID>.
Most of the testing predates me, however from archaeological expeditions
into old test servers, we’ve used JBOD configurations at some point in the
past for at least some purposes. I assume that RAID 10 was chosen
specifically for the redundancy (previously, deployments and maintenance
had been much more difficult than it is for us now), combined with the
performance aspects of going with RAID 10 when it comes to reads.
Given the progress we’ve made with automated remediation systems, it would
probably be worth another try to see if we can automate the process and
get better performance. The RAID controller itself is a continual source
of pain (a single reset can throw a cluster into chaos for a bit).
-Todd
On 7/7/14, 8:43 PM, "Bert Corderman" <be...@gmail.com> wrote:
>Thanks for the updated deck. I had not seen that one yet. I noticed in
>the preso you are running RAID10 in prod. Any thoughts of going JBOD? In
>our testing we saw significant performance improvements. This of course
>comes with trade off of manual steps if brokers fail.
>
>Bert
>
>On Monday, July 7, 2014, Todd Palino <tp...@linkedin.com.invalid> wrote:
>
>> You¹re out of date, Jun. We¹re up to 20 now :)
>>
>> Our ops presentation on Kafka is a little more up to date on numbers:
>> http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-as-a-service
>>
>> -Todd
>>
>>
>> On 7/7/14, 7:21 AM, "Jun Rao" <junrao@gmail.com <javascript:;>> wrote:
>>
>> >LinkedIn's largest Kafka cluster has 16 nodes now. More detailed info
>>can
>> >be found in
>> >
>>
>>http://www.slideshare.net/Hadoop_Summit/building-a-realtime-data-pipeline
>>-
>> >apache-kafka-at-linkedin?from_search=5
>> >
>> >Thanks,
>> >
>> >Jun
>> >
>> >
>> >On Mon, Jul 7, 2014 at 3:33 AM, Ersin Er <ersin.er@gmail.com
>> <javascript:;>> wrote:
>> >
>> >> Hi,
>> >>
>> >> LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are
>> >>larger
>> >> deployments than LinkedIn's. What's are the largest Kafka deployments
>> >>you
>> >> know of? Any public performance and scalability data published for
>>such
>> >> clusters?
>> >>
>> >> Any pointers would be interesting and helpful.
>> >>
>> >> Regards,
>> >>
>> >> --
>> >> Ersin Er
>> >>
>>
>>
Re: Largest known Kafka deployment?
Posted by Bert Corderman <be...@gmail.com>.
Thanks for the updated deck. I had not seen that one yet. I noticed in
the preso you are running RAID10 in prod. Any thoughts of going JBOD? In
our testing we saw significant performance improvements. This of course
comes with trade off of manual steps if brokers fail.
Bert
On Monday, July 7, 2014, Todd Palino <tp...@linkedin.com.invalid> wrote:
> You¹re out of date, Jun. We¹re up to 20 now :)
>
> Our ops presentation on Kafka is a little more up to date on numbers:
> http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-as-a-service
>
> -Todd
>
>
> On 7/7/14, 7:21 AM, "Jun Rao" <junrao@gmail.com <javascript:;>> wrote:
>
> >LinkedIn's largest Kafka cluster has 16 nodes now. More detailed info can
> >be found in
> >
> http://www.slideshare.net/Hadoop_Summit/building-a-realtime-data-pipeline-
> >apache-kafka-at-linkedin?from_search=5
> >
> >Thanks,
> >
> >Jun
> >
> >
> >On Mon, Jul 7, 2014 at 3:33 AM, Ersin Er <ersin.er@gmail.com
> <javascript:;>> wrote:
> >
> >> Hi,
> >>
> >> LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are
> >>larger
> >> deployments than LinkedIn's. What's are the largest Kafka deployments
> >>you
> >> know of? Any public performance and scalability data published for such
> >> clusters?
> >>
> >> Any pointers would be interesting and helpful.
> >>
> >> Regards,
> >>
> >> --
> >> Ersin Er
> >>
>
>
Re: Largest known Kafka deployment?
Posted by Todd Palino <tp...@linkedin.com.INVALID>.
You¹re out of date, Jun. We¹re up to 20 now :)
Our ops presentation on Kafka is a little more up to date on numbers:
http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-as-a-service
-Todd
On 7/7/14, 7:21 AM, "Jun Rao" <ju...@gmail.com> wrote:
>LinkedIn's largest Kafka cluster has 16 nodes now. More detailed info can
>be found in
>http://www.slideshare.net/Hadoop_Summit/building-a-realtime-data-pipeline-
>apache-kafka-at-linkedin?from_search=5
>
>Thanks,
>
>Jun
>
>
>On Mon, Jul 7, 2014 at 3:33 AM, Ersin Er <er...@gmail.com> wrote:
>
>> Hi,
>>
>> LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are
>>larger
>> deployments than LinkedIn's. What's are the largest Kafka deployments
>>you
>> know of? Any public performance and scalability data published for such
>> clusters?
>>
>> Any pointers would be interesting and helpful.
>>
>> Regards,
>>
>> --
>> Ersin Er
>>
Re: Largest known Kafka deployment?
Posted by Jun Rao <ju...@gmail.com>.
LinkedIn's largest Kafka cluster has 16 nodes now. More detailed info can
be found in
http://www.slideshare.net/Hadoop_Summit/building-a-realtime-data-pipeline-apache-kafka-at-linkedin?from_search=5
Thanks,
Jun
On Mon, Jul 7, 2014 at 3:33 AM, Ersin Er <er...@gmail.com> wrote:
> Hi,
>
> LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are larger
> deployments than LinkedIn's. What's are the largest Kafka deployments you
> know of? Any public performance and scalability data published for such
> clusters?
>
> Any pointers would be interesting and helpful.
>
> Regards,
>
> --
> Ersin Er
>