LeveledCompaction and/or SnappyCompressor causing memory pressure during repair

[Thomas van Neerijnen <to...@bossastudios.com>]

Hi all

Running Cassandra 1.0.7, I recently changed a few read heavy column
families from SizeTieredCompactionStrategy to LeveledCompactionStrategy and
added in SnappyCompressor, all with defaults so 5MB files and if memory
serves me correctly 64k chunk size for compression.
The results were amazingly good, my data size halved and my heap usage and
performance stabilised nicely, until it came time to run a repair.

When a repair isn't running I'm seeing a saw toothed pattern on my heap
graphs with CMS clearing out about 1.5GB each GC run. The CMS GC appears as
a sudden vertical drop on the Old Gen usage graph. In addition to what I
consider a healthy looking heap usage, my par new and CMS collections are
running far quicker than before I made the changes.

However, when I run a repair my CMS usage graph no longer shows sudden
drops but rather gradual slopes and only manages to clear around 300MB each
GC. This seems to occur on 2 other nodes in my cluster around the same
time, I assume this is because they're the replicas (we use 3 replicas).
Parnew collections look about the same on my graphs with or without repair
running so no trouble there so far as I can tell.
The symptom of the memory pressure during repair is either the node running
the repair of one of the two replicas tends to perform badly with read
stage backing up into the thousands at times.
If I run a repair on more than one or two nodes at the same time (it's a 7
node cluster) the memory pressure is so bad that half the cluster ends up
OOMing, and this happened during off-peak when it's doing about half the
reads we handle during peak so not particularly loaded.

The question I'm asking is has anyone run into this behaviour before, and
if so how was it dealt with?

Once I have nursed the cluster thru the repair it's currently running I
will be turning off compression on one of my larger CFs to see if it makes
a difference, I'll send the results of that test tomorrow.

Re: LeveledCompaction and/or SnappyCompressor causing memory pressure during repair

[Thomas van Neerijnen <to...@bossastudios.com>]

Thanks for the suggestions but I'd already removed the compression when
your message came thru. That alleviated the problem but didn't solve it.
I'm still looking at a few other possible causes, I'll post back if I work
out what's going on, for now I am running rolling repairs to avoid another
outage.

On Sun, Mar 11, 2012 at 6:32 PM, Edward Capriolo <ed...@gmail.com>wrote:

> One thing you may want to look at is the meanRowSize from nodetool
> cfstats and your compression block size. In our case the mean
> compacted size is 560 bytes and 64KB block size caused CPU tickets and
> a lot of short lived memory. I have brought by block size down to 16K.
> The result tables are not noticeably larger and less memory pressure
> on the young gen. I might try going down to 4 K next.
>
> On Sat, Mar 10, 2012 at 5:38 PM, Edward Capriolo <ed...@gmail.com>
> wrote:
> > The only downside of compression is it does cause more memory
> > pressure. I can imagine something like repair could confound this.
> > Since it would seem like building the merkle tree would involve
> > decompressing every block on disk.
> >
> > I have been attempting to determine if the block size being larger or
> > smaller has any effect on memory pressure.
> >
> > On Sat, Mar 10, 2012 at 4:50 PM, Peter Schuller
> > <pe...@infidyne.com> wrote:
> >>> However, when I run a repair my CMS usage graph no longer shows sudden
> drops
> >>> but rather gradual slopes and only manages to clear around 300MB each
> GC.
> >>> This seems to occur on 2 other nodes in my cluster around the same
> time, I
> >>> assume this is because they're the replicas (we use 3 replicas). Parnew
> >>> collections look about the same on my graphs with or without repair
> running
> >>> so no trouble there so far as I can tell.
> >>
> >> I don't know why leveled/snappy would affect it, but disregarding
> >> that, I would have been suggesting that you are seeing additional heap
> >> usage because of long-running repairs retaining sstables and delaying
> >> their unload/removal (index sampling/bloom filters filling your heap).
> >> If it really only happens for leveled/snappy however, I don't know
> >> what that might be caused by.
> >>
> >> --
> >> / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
>

Re: LeveledCompaction and/or SnappyCompressor causing memory pressure during repair

[Edward Capriolo <ed...@gmail.com>]

One thing you may want to look at is the meanRowSize from nodetool
cfstats and your compression block size. In our case the mean
compacted size is 560 bytes and 64KB block size caused CPU tickets and
a lot of short lived memory. I have brought by block size down to 16K.
The result tables are not noticeably larger and less memory pressure
on the young gen. I might try going down to 4 K next.

On Sat, Mar 10, 2012 at 5:38 PM, Edward Capriolo <ed...@gmail.com> wrote:
> The only downside of compression is it does cause more memory
> pressure. I can imagine something like repair could confound this.
> Since it would seem like building the merkle tree would involve
> decompressing every block on disk.
>
> I have been attempting to determine if the block size being larger or
> smaller has any effect on memory pressure.
>
> On Sat, Mar 10, 2012 at 4:50 PM, Peter Schuller
> <pe...@infidyne.com> wrote:
>>> However, when I run a repair my CMS usage graph no longer shows sudden drops
>>> but rather gradual slopes and only manages to clear around 300MB each GC.
>>> This seems to occur on 2 other nodes in my cluster around the same time, I
>>> assume this is because they're the replicas (we use 3 replicas). Parnew
>>> collections look about the same on my graphs with or without repair running
>>> so no trouble there so far as I can tell.
>>
>> I don't know why leveled/snappy would affect it, but disregarding
>> that, I would have been suggesting that you are seeing additional heap
>> usage because of long-running repairs retaining sstables and delaying
>> their unload/removal (index sampling/bloom filters filling your heap).
>> If it really only happens for leveled/snappy however, I don't know
>> what that might be caused by.
>>
>> --
>> / Peter Schuller (@scode, http://worldmodscode.wordpress.com)

Re: LeveledCompaction and/or SnappyCompressor causing memory pressure during repair

[Edward Capriolo <ed...@gmail.com>]

The only downside of compression is it does cause more memory
pressure. I can imagine something like repair could confound this.
Since it would seem like building the merkle tree would involve
decompressing every block on disk.

I have been attempting to determine if the block size being larger or
smaller has any effect on memory pressure.

On Sat, Mar 10, 2012 at 4:50 PM, Peter Schuller
<pe...@infidyne.com> wrote:
>> However, when I run a repair my CMS usage graph no longer shows sudden drops
>> but rather gradual slopes and only manages to clear around 300MB each GC.
>> This seems to occur on 2 other nodes in my cluster around the same time, I
>> assume this is because they're the replicas (we use 3 replicas). Parnew
>> collections look about the same on my graphs with or without repair running
>> so no trouble there so far as I can tell.
>
> I don't know why leveled/snappy would affect it, but disregarding
> that, I would have been suggesting that you are seeing additional heap
> usage because of long-running repairs retaining sstables and delaying
> their unload/removal (index sampling/bloom filters filling your heap).
> If it really only happens for leveled/snappy however, I don't know
> what that might be caused by.
>
> --
> / Peter Schuller (@scode, http://worldmodscode.wordpress.com)

Re: LeveledCompaction and/or SnappyCompressor causing memory pressure during repair

[Peter Schuller <pe...@infidyne.com>]

> However, when I run a repair my CMS usage graph no longer shows sudden drops
> but rather gradual slopes and only manages to clear around 300MB each GC.
> This seems to occur on 2 other nodes in my cluster around the same time, I
> assume this is because they're the replicas (we use 3 replicas). Parnew
> collections look about the same on my graphs with or without repair running
> so no trouble there so far as I can tell.

I don't know why leveled/snappy would affect it, but disregarding
that, I would have been suggesting that you are seeing additional heap
usage because of long-running repairs retaining sstables and delaying
their unload/removal (index sampling/bloom filters filling your heap).
If it really only happens for leveled/snappy however, I don't know
what that might be caused by.

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)