You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dan Washusen <da...@reactive.org> on 2011/03/25 02:52:03 UTC
OOM during compaction with half the heap still available?
Hey All,
I've noticed that the Cassandra instance I have running on our build machine occasionally crashes with an OOM error during compaction. I'm going to dial down the memtable thresholds etc but I was wondering if anyone could help explain the heap usage at the time of the crash. I just happened to leave a JMX console window open and it's showing that just before the crash roughly 50% of the heap was still available.
Screenshot of heap usage: http://img3.imageshack.us/img3/2822/memoryf.png
The above screenshot was taken a few weeks ago on Cassandra 0.7.2 (I think) with nmap disabled on Java 1.6.0_24-b07. I'm now running 0.7.4 with nmap enabled and just got the same crash (based on the error message in the log)...
Log snippet from crash that matches screenshot: http://pastebin.com/ACa8fKUu
Log snippet from 0.7.4 crash: http://pastebin.com/SwSQawUM
Cheers,
--
Dan Washusen
Re: OOM during compaction with half the heap still available?
Posted by Dan Washusen <da...@reactive.org>.
Ah, it would appear I forgot to do that on the hudson machine. Thanks!
--
Dan Washusen
On Friday, 25 March 2011 at 2:23 PM, Jonathan Ellis wrote:
> Have you run nodetool scrub? The data versioning problem scrub fixes
> can manifest itself as trying to read GB of data into memory during
> compaction.
>
> On Thu, Mar 24, 2011 at 8:52 PM, Dan Washusen <da...@reactive.org> wrote:
> > Hey All,
> > I've noticed that the Cassandra instance I have running on our build machine
> > occasionally crashes with an OOM error during compaction. I'm going to dial
> > down the memtable thresholds etc but I was wondering if anyone could help
> > explain the heap usage at the time of the crash. I just happened to leave a
> > JMX console window open and it's showing that just before the crash roughly
> > 50% of the heap was still available.
> > Screenshot of heap usage: http://img3.imageshack.us/img3/2822/memoryf.png
> > The above screenshot was taken a few weeks ago on Cassandra 0.7.2 (I think)
> > with nmap disabled on Java 1.6.0_24-b07. I'm now running 0.7.4 with nmap
> > enabled and just got the same crash (based on the error message in the
> > log)...
> > Log snippet from crash that matches screenshot: http://pastebin.com/ACa8fKUu
> > Log snippet from 0.7.4 crash: http://pastebin.com/SwSQawUM
> > Cheers,
> > --
> > Dan Washusen
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: OOM during compaction with half the heap still available?
Posted by Jonathan Ellis <jb...@gmail.com>.
Have you run nodetool scrub? The data versioning problem scrub fixes
can manifest itself as trying to read GB of data into memory during
compaction.
On Thu, Mar 24, 2011 at 8:52 PM, Dan Washusen <da...@reactive.org> wrote:
> Hey All,
> I've noticed that the Cassandra instance I have running on our build machine
> occasionally crashes with an OOM error during compaction. I'm going to dial
> down the memtable thresholds etc but I was wondering if anyone could help
> explain the heap usage at the time of the crash. I just happened to leave a
> JMX console window open and it's showing that just before the crash roughly
> 50% of the heap was still available.
> Screenshot of heap usage: http://img3.imageshack.us/img3/2822/memoryf.png
> The above screenshot was taken a few weeks ago on Cassandra 0.7.2 (I think)
> with nmap disabled on Java 1.6.0_24-b07. I'm now running 0.7.4 with nmap
> enabled and just got the same crash (based on the error message in the
> log)...
> Log snippet from crash that matches screenshot: http://pastebin.com/ACa8fKUu
> Log snippet from 0.7.4 crash: http://pastebin.com/SwSQawUM
> Cheers,
> --
> Dan Washusen
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com