You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Chris Burroughs <ch...@gmail.com> on 2011/02/02 15:22:14 UTC
Re: reduced cached mem; resident set size growth
On 01/28/2011 09:19 PM, Chris Burroughs wrote:
> Thanks Oleg and Zhu. I swear that wasn't a new hotspot version when I
> checked, but that's obviously not the case. I'll update one node to the
> latest as soon as I can and report back.
RSS over 48 hours with java 6 update 23:
http://img716.imageshack.us/img716/5202/u2348hours.png
I'll continue monitoring but RSS still appears to grow without bounds.
Zhu reported a similar problem with Ubuntu 10.04. While possible, it
would seem seam extraordinary unlikely that there is a glibc or kernel
bug affecting us both.
Re: reduced cached mem; resident set size growth
Posted by Ryan King <ry...@twitter.com>.
The test was inconclusive because we decomissioned that cluster before
it'd be running long enough to exhibit the problem.
-ryan
On Wed, Mar 16, 2011 at 7:27 PM, Zhu Han <sc...@gmail.com> wrote:
>
>
> On Thu, Feb 3, 2011 at 1:49 AM, Ryan King <ry...@twitter.com> wrote:
>>
>> On Wed, Feb 2, 2011 at 6:22 AM, Chris Burroughs
>> <ch...@gmail.com> wrote:
>> > On 01/28/2011 09:19 PM, Chris Burroughs wrote:
>> >> Thanks Oleg and Zhu. I swear that wasn't a new hotspot version when I
>> >> checked, but that's obviously not the case. I'll update one node to
>> >> the
>> >> latest as soon as I can and report back.
>> >
>> >
>> > RSS over 48 hours with java 6 update 23:
>> >
>> > http://img716.imageshack.us/img716/5202/u2348hours.png
>> >
>> > I'll continue monitoring but RSS still appears to grow without bounds.
>> > Zhu reported a similar problem with Ubuntu 10.04. While possible, it
>> > would seem seam extraordinary unlikely that there is a glibc or kernel
>> > bug affecting us both.
>>
>> We're seeing a similar problem with one of our clusters (but over a
>> longer time scale). Its possible that its not a leak, but just
>> fragmentation. Unless you've told it otherwise, the jvm uses glibc's
>> malloc implementation for off-heap allocations. We're currently
>> running a test with jemalloc on one node to see if the problem goes
>> away.
>
> Ryan, does jemalloc solve the RSS growth problem in your test?
>
>> -ryan
>
>
Re: reduced cached mem; resident set size growth
Posted by Zhu Han <sc...@gmail.com>.
On Thu, Mar 17, 2011 at 10:27 AM, Zhu Han <sc...@gmail.com> wrote:
>
>
> On Thu, Feb 3, 2011 at 1:49 AM, Ryan King <ry...@twitter.com> wrote:
>
>> On Wed, Feb 2, 2011 at 6:22 AM, Chris Burroughs
>> <ch...@gmail.com> wrote:
>> > On 01/28/2011 09:19 PM, Chris Burroughs wrote:
>> >> Thanks Oleg and Zhu. I swear that wasn't a new hotspot version when I
>> >> checked, but that's obviously not the case. I'll update one node to
>> the
>> >> latest as soon as I can and report back.
>> >
>> >
>> > RSS over 48 hours with java 6 update 23:
>> >
>> > http://img716.imageshack.us/img716/5202/u2348hours.png
>> >
>> > I'll continue monitoring but RSS still appears to grow without bounds.
>> > Zhu reported a similar problem with Ubuntu 10.04. While possible, it
>> > would seem seam extraordinary unlikely that there is a glibc or kernel
>> > bug affecting us both.
>>
>> We're seeing a similar problem with one of our clusters (but over a
>> longer time scale).
>
>
Does it mean not all your clusters running cassandra observed the same RSS
growth problem?
> Its possible that its not a leak, but just
>> fragmentation. Unless you've told it otherwise, the jvm uses glibc's
>> malloc implementation for off-heap allocations. We're currently
>> running a test with jemalloc on one node to see if the problem goes
>> away.
>>
>
> Ryan, does jemalloc solve the RSS growth problem in your test?
>
> -ryan
>>
>
>
Re: reduced cached mem; resident set size growth
Posted by Zhu Han <sc...@gmail.com>.
On Thu, Feb 3, 2011 at 1:49 AM, Ryan King <ry...@twitter.com> wrote:
> On Wed, Feb 2, 2011 at 6:22 AM, Chris Burroughs
> <ch...@gmail.com> wrote:
> > On 01/28/2011 09:19 PM, Chris Burroughs wrote:
> >> Thanks Oleg and Zhu. I swear that wasn't a new hotspot version when I
> >> checked, but that's obviously not the case. I'll update one node to the
> >> latest as soon as I can and report back.
> >
> >
> > RSS over 48 hours with java 6 update 23:
> >
> > http://img716.imageshack.us/img716/5202/u2348hours.png
> >
> > I'll continue monitoring but RSS still appears to grow without bounds.
> > Zhu reported a similar problem with Ubuntu 10.04. While possible, it
> > would seem seam extraordinary unlikely that there is a glibc or kernel
> > bug affecting us both.
>
> We're seeing a similar problem with one of our clusters (but over a
> longer time scale). Its possible that its not a leak, but just
> fragmentation. Unless you've told it otherwise, the jvm uses glibc's
> malloc implementation for off-heap allocations. We're currently
> running a test with jemalloc on one node to see if the problem goes
> away.
>
Ryan, does jemalloc solve the RSS growth problem in your test?
-ryan
>
Re: reduced cached mem; resident set size growth
Posted by Ryan King <ry...@twitter.com>.
On Wed, Feb 2, 2011 at 10:29 AM, Chris Burroughs
<ch...@gmail.com> wrote:
> On 02/02/2011 12:49 PM, Ryan King wrote:
>> We're seeing a similar problem with one of our clusters (but over a
>> longer time scale). Its possible that its not a leak, but just
>> fragmentation. Unless you've told it otherwise, the jvm uses glibc's
>> malloc implementation for off-heap allocations. We're currently
>> running a test with jemalloc on one node to see if the problem goes
>> away.
>>
>
> Thanks Ryan.
>
> Is it over a longer time scale because of some action taken to mitigate
> the problem, or has it always been that long for you?
My guess is that its a longer timeframe because the cluster is really
low traffic (around 100qps across 10 nodes).
-ryan
Re: reduced cached mem; resident set size growth
Posted by Chris Burroughs <ch...@gmail.com>.
On 02/02/2011 12:49 PM, Ryan King wrote:
> We're seeing a similar problem with one of our clusters (but over a
> longer time scale). Its possible that its not a leak, but just
> fragmentation. Unless you've told it otherwise, the jvm uses glibc's
> malloc implementation for off-heap allocations. We're currently
> running a test with jemalloc on one node to see if the problem goes
> away.
>
Thanks Ryan.
Is it over a longer time scale because of some action taken to mitigate
the problem, or has it always been that long for you?
Re: reduced cached mem; resident set size growth
Posted by Ryan King <ry...@twitter.com>.
On Wed, Feb 2, 2011 at 6:22 AM, Chris Burroughs
<ch...@gmail.com> wrote:
> On 01/28/2011 09:19 PM, Chris Burroughs wrote:
>> Thanks Oleg and Zhu. I swear that wasn't a new hotspot version when I
>> checked, but that's obviously not the case. I'll update one node to the
>> latest as soon as I can and report back.
>
>
> RSS over 48 hours with java 6 update 23:
>
> http://img716.imageshack.us/img716/5202/u2348hours.png
>
> I'll continue monitoring but RSS still appears to grow without bounds.
> Zhu reported a similar problem with Ubuntu 10.04. While possible, it
> would seem seam extraordinary unlikely that there is a glibc or kernel
> bug affecting us both.
We're seeing a similar problem with one of our clusters (but over a
longer time scale). Its possible that its not a leak, but just
fragmentation. Unless you've told it otherwise, the jvm uses glibc's
malloc implementation for off-heap allocations. We're currently
running a test with jemalloc on one node to see if the problem goes
away.
-ryan