You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Zakee <kz...@netzero.net> on 2015/02/19 01:13:19 UTC
Broker w/ high memory due to index file sizes
I am running a cluster of 5 brokers with 40G ms/mx for each. I found one of
the brokers is constantly using above ~90% of memory for jvm.heapUsage. I
checked from lsof output that the size of the index files for this broker
is too large.
Not sure what is going on with this one broker in the cluster? Why would
the index file sizes be so hugely different on one broker? Any ideas?
Regards
Zakee
____________________________________________________________
Invest with the Trend
Exclusive Breakout Alert On Soaring Social Media Technology
http://thirdpartyoffers.netzero.net/TGL3231/54e52a9fe121d2a9f4a27st01vuc
Re: Broker w/ high memory due to index file sizes
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
I'd start small and grow if needed.... in general. There is one bit of
info missing here. Did you try with a smaller heap and had issues? If so,
which issues? A few references from our internal Wiki:
- XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-Xloggc:/var/log/gc/gc.log-$$-$(hostname)-$(date +'%Y%m%d%H%M').log
- https://github.com/chewiebug/GCViewer
- Good general JVM info:
http://blog.asquareb.com/blog/2014/12/13/jvm-gc-settings-and-hbase/
- Good general JVM info:
https://blog.codecentric.de/en/2012/08/useful-jvm-flags-part-5-young-generation-garbage-collection/
- Lower Xmx to <32GB heap because of pointer compression.
https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Thu, Feb 19, 2015 at 1:53 PM, Zakee <kz...@netzero.net> wrote:
> Well are there any measurement techniques for Memory config in brokers. We
> do have a large load, with a max throughput 200MB/s. What do you suggest as
> the recommended memory config for 5 brokers to handle such loads?
>
> On Wed, Feb 18, 2015 at 7:13 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > 40G is really huge, generally you would want more like 4G. Are you sure
> you
> > need that? Not sure what you mean by lsof and index files being too
> large,
> > but the index files are memory mapped so they should be able to grow
> > arbitrarily large and their memory usage is not counted in the java heap
> > (in fact by having such a large heap you are taking away OS memory from
> > them).
> >
> > -Jay
> >
> > On Wed, Feb 18, 2015 at 4:13 PM, Zakee <kz...@netzero.net> wrote:
> >
> > > I am running a cluster of 5 brokers with 40G ms/mx for each. I found
> one
> > of
> > > the brokers is constantly using above ~90% of memory for
> jvm.heapUsage. I
> > > checked from lsof output that the size of the index files for this
> broker
> > > is too large.
> > >
> > > Not sure what is going on with this one broker in the cluster? Why
> would
> > > the index file sizes be so hugely different on one broker? Any ideas?
> > >
> > >
> > > Regards
> > > Zakee
> > > ____________________________________________________________
> > > Invest with the Trend
> > > Exclusive Breakout Alert On Soaring Social Media Technology
> > >
> http://thirdpartyoffers.netzero.net/TGL3231/54e52a9fe121d2a9f4a27st01vuc
> > ____________________________________________________________
> > Have you been injured?
> > Get a free evaluation today to see what your injury case is worth.
> > http://thirdpartyoffers.netzero.net/TGL3255/54e55ad9894265ad90bcbmp13duc
>
Re: Broker w/ high memory due to index file sizes
Posted by Zakee <kz...@netzero.net>.
Thanks, Ewen. I will try these.
-Zakee
On Sun, Feb 22, 2015 at 11:51 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> If you haven't seen it yet, you probably want to look at
> http://kafka.apache.org/documentation.html#java
>
> -Ewen
>
> On Thu, Feb 19, 2015 at 10:53 AM, Zakee <kz...@netzero.net> wrote:
>
> > Well are there any measurement techniques for Memory config in brokers.
> We
> > do have a large load, with a max throughput 200MB/s. What do you suggest
> as
> > the recommended memory config for 5 brokers to handle such loads?
> >
> > On Wed, Feb 18, 2015 at 7:13 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > 40G is really huge, generally you would want more like 4G. Are you sure
> > you
> > > need that? Not sure what you mean by lsof and index files being too
> > large,
> > > but the index files are memory mapped so they should be able to grow
> > > arbitrarily large and their memory usage is not counted in the java
> heap
> > > (in fact by having such a large heap you are taking away OS memory from
> > > them).
> > >
> > > -Jay
> > >
> > > On Wed, Feb 18, 2015 at 4:13 PM, Zakee <kz...@netzero.net> wrote:
> > >
> > > > I am running a cluster of 5 brokers with 40G ms/mx for each. I found
> > one
> > > of
> > > > the brokers is constantly using above ~90% of memory for
> > jvm.heapUsage. I
> > > > checked from lsof output that the size of the index files for this
> > broker
> > > > is too large.
> > > >
> > > > Not sure what is going on with this one broker in the cluster? Why
> > would
> > > > the index file sizes be so hugely different on one broker? Any ideas?
> > > >
> > > >
> > > > Regards
> > > > Zakee
> > > > ____________________________________________________________
> > > > Invest with the Trend
> > > > Exclusive Breakout Alert On Soaring Social Media Technology
> > > >
> > http://thirdpartyoffers.netzero.net/TGL3231/54e52a9fe121d2a9f4a27st01vuc
> > > ____________________________________________________________
> > > Have you been injured?
> > > Get a free evaluation today to see what your injury case is worth.
> > >
> http://thirdpartyoffers.netzero.net/TGL3255/54e55ad9894265ad90bcbmp13duc
> >
>
>
>
> --
> Thanks,
> Ewen
> ____________________________________________________________
> High School Yearbooks
> View Class Yearbooks Online Free. Reminisce & Buy a Reprint Today!
> http://thirdpartyoffers.netzero.net/TGL3255/54eae3c81e2aa63c86598mp11duc
Re: Broker w/ high memory due to index file sizes
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
If you haven't seen it yet, you probably want to look at
http://kafka.apache.org/documentation.html#java
-Ewen
On Thu, Feb 19, 2015 at 10:53 AM, Zakee <kz...@netzero.net> wrote:
> Well are there any measurement techniques for Memory config in brokers. We
> do have a large load, with a max throughput 200MB/s. What do you suggest as
> the recommended memory config for 5 brokers to handle such loads?
>
> On Wed, Feb 18, 2015 at 7:13 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > 40G is really huge, generally you would want more like 4G. Are you sure
> you
> > need that? Not sure what you mean by lsof and index files being too
> large,
> > but the index files are memory mapped so they should be able to grow
> > arbitrarily large and their memory usage is not counted in the java heap
> > (in fact by having such a large heap you are taking away OS memory from
> > them).
> >
> > -Jay
> >
> > On Wed, Feb 18, 2015 at 4:13 PM, Zakee <kz...@netzero.net> wrote:
> >
> > > I am running a cluster of 5 brokers with 40G ms/mx for each. I found
> one
> > of
> > > the brokers is constantly using above ~90% of memory for
> jvm.heapUsage. I
> > > checked from lsof output that the size of the index files for this
> broker
> > > is too large.
> > >
> > > Not sure what is going on with this one broker in the cluster? Why
> would
> > > the index file sizes be so hugely different on one broker? Any ideas?
> > >
> > >
> > > Regards
> > > Zakee
> > > ____________________________________________________________
> > > Invest with the Trend
> > > Exclusive Breakout Alert On Soaring Social Media Technology
> > >
> http://thirdpartyoffers.netzero.net/TGL3231/54e52a9fe121d2a9f4a27st01vuc
> > ____________________________________________________________
> > Have you been injured?
> > Get a free evaluation today to see what your injury case is worth.
> > http://thirdpartyoffers.netzero.net/TGL3255/54e55ad9894265ad90bcbmp13duc
>
--
Thanks,
Ewen
Re: Broker w/ high memory due to index file sizes
Posted by Zakee <kz...@netzero.net>.
Well are there any measurement techniques for Memory config in brokers. We
do have a large load, with a max throughput 200MB/s. What do you suggest as
the recommended memory config for 5 brokers to handle such loads?
On Wed, Feb 18, 2015 at 7:13 PM, Jay Kreps <ja...@gmail.com> wrote:
> 40G is really huge, generally you would want more like 4G. Are you sure you
> need that? Not sure what you mean by lsof and index files being too large,
> but the index files are memory mapped so they should be able to grow
> arbitrarily large and their memory usage is not counted in the java heap
> (in fact by having such a large heap you are taking away OS memory from
> them).
>
> -Jay
>
> On Wed, Feb 18, 2015 at 4:13 PM, Zakee <kz...@netzero.net> wrote:
>
> > I am running a cluster of 5 brokers with 40G ms/mx for each. I found one
> of
> > the brokers is constantly using above ~90% of memory for jvm.heapUsage. I
> > checked from lsof output that the size of the index files for this broker
> > is too large.
> >
> > Not sure what is going on with this one broker in the cluster? Why would
> > the index file sizes be so hugely different on one broker? Any ideas?
> >
> >
> > Regards
> > Zakee
> > ____________________________________________________________
> > Invest with the Trend
> > Exclusive Breakout Alert On Soaring Social Media Technology
> > http://thirdpartyoffers.netzero.net/TGL3231/54e52a9fe121d2a9f4a27st01vuc
> ____________________________________________________________
> Have you been injured?
> Get a free evaluation today to see what your injury case is worth.
> http://thirdpartyoffers.netzero.net/TGL3255/54e55ad9894265ad90bcbmp13duc
Re: Broker w/ high memory due to index file sizes
Posted by Jay Kreps <ja...@gmail.com>.
40G is really huge, generally you would want more like 4G. Are you sure you
need that? Not sure what you mean by lsof and index files being too large,
but the index files are memory mapped so they should be able to grow
arbitrarily large and their memory usage is not counted in the java heap
(in fact by having such a large heap you are taking away OS memory from
them).
-Jay
On Wed, Feb 18, 2015 at 4:13 PM, Zakee <kz...@netzero.net> wrote:
> I am running a cluster of 5 brokers with 40G ms/mx for each. I found one of
> the brokers is constantly using above ~90% of memory for jvm.heapUsage. I
> checked from lsof output that the size of the index files for this broker
> is too large.
>
> Not sure what is going on with this one broker in the cluster? Why would
> the index file sizes be so hugely different on one broker? Any ideas?
>
>
> Regards
> Zakee
> ____________________________________________________________
> Invest with the Trend
> Exclusive Breakout Alert On Soaring Social Media Technology
> http://thirdpartyoffers.netzero.net/TGL3231/54e52a9fe121d2a9f4a27st01vuc