You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jennifer Fountain <jf...@meetme.com> on 2015/10/26 16:32:04 UTC
Kafka Best Practices in AWS
We installed four nodes (r3.xlarge) with approx 100 topics. Some topics
have a replication factor of 4, some have 3 and some have 2. Each node is
running about 90% cpu usage and we are seeing performance issues with
messages consumption.
Would anyone have any words of wisdom or recommendations for AWS? Should
we add more nodes? What should be the replication factor for x amount of
nodes?
Thank you!
--
Jennifer Fountain
DevOPS
Re: Kafka Best Practices in AWS
Posted by Jennifer Fountain <jf...@meetme.com>.
On Mon, Oct 26, 2015 at 1:18 PM, Steve Brandon <st...@sparkcentral.com>
wrote:
> Depending on the consumer, I tend to work with c4 types and have found
> better success. Although we run about 200+ consumers per server on 16 core
> servers and see moderate load. YMMV
>
We are currently using storm for our consumer. What would you recommend
the number of brokers vs replication factor? What is the best way to scale
out?
replication factor 3 or 4? 4 nodes or 6 nodes?
Thanks!
On Oct 26, 2015 8:32 AM, "Jennifer Fountain" <jf...@meetme.com> wrote:
>
> > We installed four nodes (r3.xlarge) with approx 100 topics. Some topics
> > have a replication factor of 4, some have 3 and some have 2. Each node is
> > running about 90% cpu usage and we are seeing performance issues with
> > messages consumption.
> >
> > Would anyone have any words of wisdom or recommendations for AWS? Should
> > we add more nodes? What should be the replication factor for x amount of
> > nodes?
> >
> > Thank you!
> >
> > --
> >
> >
> > Jennifer Fountain
> > DevOPS
> >
>
--
Jennifer Fountain
DevOPS
Re: Kafka Best Practices in AWS
Posted by Steve Brandon <st...@sparkcentral.com>.
Depending on the consumer, I tend to work with c4 types and have found
better success. Although we run about 200+ consumers per server on 16 core
servers and see moderate load. YMMV
On Oct 26, 2015 8:32 AM, "Jennifer Fountain" <jf...@meetme.com> wrote:
> We installed four nodes (r3.xlarge) with approx 100 topics. Some topics
> have a replication factor of 4, some have 3 and some have 2. Each node is
> running about 90% cpu usage and we are seeing performance issues with
> messages consumption.
>
> Would anyone have any words of wisdom or recommendations for AWS? Should
> we add more nodes? What should be the replication factor for x amount of
> nodes?
>
> Thank you!
>
> --
>
>
> Jennifer Fountain
> DevOPS
>