Understanding java options used by Kafka

[Marc Labbe <mr...@gmail.com>]

Hi,

I am fairly new to java and I am trying to figure out the reason behind
each option listed on http://kafka.apache.org/documentation.html#java.
Sorry if this a noob question :)

In this list, one of option particularly struck my attention and had me
wondering if the whole list of options is correct:

-Djava.awt.headless=true

My understanding is the AWT is a toolkit for UI applications, which is not
really the case of Kafka. Is this correct?

Also, is it possible to have more information on why some specific GC
options are used at LinkedIn? Is there anything in there specifically
helping Kafka's performance?

I am thinking of NewSize, MaxNewSixze, UseConcMarkSweepGC and
CMSInitiatingOccupancyFraction.

I looked them up and found what they are but is there an explanation about
why/if we should use them with Kafka?

thanks

Re: Understanding java options used by Kafka

[Marc Labbe <mr...@gmail.com>]

Thanks Neha, it does help. I am running tests without those settings, and
I'll run them again with it with gc monitoring.
On Sep 23, 2013 12:02 PM, "Neha Narkhede" <ne...@gmail.com> wrote:

> I'm not sure about the reason for -Djava.awt.headless=true, but the GC
> options were picked after quite a lot of GC tuning for the client/server
> configuration that LinkedIn has deployed in production. This works well
> when producers are deployed with request.required.acks=1 and consumers with
> fetch.wait.max.ms set to default value (100 ms). The garbage collection
> settings are optimized for preventing objects from getting promoted to old
> generation. An option that helped in reducing the promotion rate is
> MaxTenuringThreshold, which is set to max (15 or 31 depending on your JVM).
> We found that sizing the young generation and survivor spaces small enough
> with a small heap worked better than having a large young gen and large
> heap, in terms of the max GC pause.
>
> Hope that helps.
>
> Thanks,
> Neha
>
>
> On Fri, Sep 20, 2013 at 12:40 PM, Marc Labbe <mr...@gmail.com> wrote:
>
> > Hi,
> >
> > I am fairly new to java and I am trying to figure out the reason behind
> > each option listed on http://kafka.apache.org/documentation.html#java.
> > Sorry if this a noob question :)
> >
> > In this list, one of option particularly struck my attention and had me
> > wondering if the whole list of options is correct:
> >
> > -Djava.awt.headless=true
> >
> > My understanding is the AWT is a toolkit for UI applications, which is
> not
> > really the case of Kafka. Is this correct?
> >
> > Also, is it possible to have more information on why some specific GC
> > options are used at LinkedIn? Is there anything in there specifically
> > helping Kafka's performance?
> >
> > I am thinking of NewSize, MaxNewSixze, UseConcMarkSweepGC and
> > CMSInitiatingOccupancyFraction.
> >
> > I looked them up and found what they are but is there an explanation
> about
> > why/if we should use them with Kafka?
> >
> > thanks
> >
>

Re: Understanding java options used by Kafka

[Neha Narkhede <ne...@gmail.com>]

I'm not sure about the reason for -Djava.awt.headless=true, but the GC
options were picked after quite a lot of GC tuning for the client/server
configuration that LinkedIn has deployed in production. This works well
when producers are deployed with request.required.acks=1 and consumers with
fetch.wait.max.ms set to default value (100 ms). The garbage collection
settings are optimized for preventing objects from getting promoted to old
generation. An option that helped in reducing the promotion rate is
MaxTenuringThreshold, which is set to max (15 or 31 depending on your JVM).
We found that sizing the young generation and survivor spaces small enough
with a small heap worked better than having a large young gen and large
heap, in terms of the max GC pause.

Hope that helps.

Thanks,
Neha


On Fri, Sep 20, 2013 at 12:40 PM, Marc Labbe <mr...@gmail.com> wrote:

> Hi,
>
> I am fairly new to java and I am trying to figure out the reason behind
> each option listed on http://kafka.apache.org/documentation.html#java.
> Sorry if this a noob question :)
>
> In this list, one of option particularly struck my attention and had me
> wondering if the whole list of options is correct:
>
> -Djava.awt.headless=true
>
> My understanding is the AWT is a toolkit for UI applications, which is not
> really the case of Kafka. Is this correct?
>
> Also, is it possible to have more information on why some specific GC
> options are used at LinkedIn? Is there anything in there specifically
> helping Kafka's performance?
>
> I am thinking of NewSize, MaxNewSixze, UseConcMarkSweepGC and
> CMSInitiatingOccupancyFraction.
>
> I looked them up and found what they are but is there an explanation about
> why/if we should use them with Kafka?
>
> thanks
>