java, oom, gc stop the world

[S Ahmed <sa...@gmail.com>]

In the wild, what sort of memory usage patterns have you guys seen with
kafka?

I'm not that well versed with java and its memory/gc issues, what have you
guys experienced with your production brokers?  (assuming consumers and
producers are not that much of an issue).

How do you determine how much memory to allocate to a given broker?  Is it
based on factors such as:

1. producers are mostly using batch versus single messages
2. messages per second
3. size of message payload
4. # of consumers connecting to a given broker
etc.

Is there a "magic" forumula?

Re: java, oom, gc stop the world

[David DeMaagd <dd...@linkedin.com>]

It's worth noting that we currently run kakfa at LinkedIn with a 5G heap 
(not 3G, still using the CMS GC though - should update that), and the 
info on that wiki is aimed at 0.7.  

We are actively working on things for 0.8 - don't have a 'this works for 
us', much less a 'recommendedation' there (hopefully will have something 
on that side soon). 

Every thing else on that wiki looks to be in the right ballpark. 

-- 
Dave DeMaagd
ddemaagd@linkedin.com | 818 262 7958

(jay.kreps@gmail.com - Tue, Jan 22, 2013 at 07:35:17PM -0800)
> We put a lot of info here:
> https://cwiki.apache.org/confluence/display/KAFKA/Operations
> 
> Does that help?
> 
> -Jay
> 
> 
> On Tue, Jan 22, 2013 at 7:14 PM, S Ahmed <sa...@gmail.com> wrote:
> 
> > In the wild, what sort of memory usage patterns have you guys seen with
> > kafka?
> >
> > I'm not that well versed with java and its memory/gc issues, what have you
> > guys experienced with your production brokers?  (assuming consumers and
> > producers are not that much of an issue).
> >
> > How do you determine how much memory to allocate to a given broker?  Is it
> > based on factors such as:
> >
> > 1. producers are mostly using batch versus single messages
> > 2. messages per second
> > 3. size of message payload
> > 4. # of consumers connecting to a given broker
> > etc.
> >
> > Is there a "magic" forumula?
> >

Re: java, oom, gc stop the world

[Jay Kreps <ja...@gmail.com>]

We put a lot of info here:
https://cwiki.apache.org/confluence/display/KAFKA/Operations

Does that help?

-Jay


On Tue, Jan 22, 2013 at 7:14 PM, S Ahmed <sa...@gmail.com> wrote:

> In the wild, what sort of memory usage patterns have you guys seen with
> kafka?
>
> I'm not that well versed with java and its memory/gc issues, what have you
> guys experienced with your production brokers?  (assuming consumers and
> producers are not that much of an issue).
>
> How do you determine how much memory to allocate to a given broker?  Is it
> based on factors such as:
>
> 1. producers are mostly using batch versus single messages
> 2. messages per second
> 3. size of message payload
> 4. # of consumers connecting to a given broker
> etc.
>
> Is there a "magic" forumula?
>