You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dan Retzlaff <dr...@gmail.com> on 2011/03/19 01:11:33 UTC
0.6.5 OOM during high read load
Dear experts, :)
Our application triggered an OOM error in Cassandra 0.6.5 by reading the
same 1.7MB column repeatedly (~80k reads). I analyzed the heap dump, and it
looks like the column value was queued 5400 times in an
OutboundTcpConnection destined for the Cassandra instance that received the
client request. Unfortunately, this intra-node connection goes across a
100Mb data center interconnect, so it was only a matter of time before the
heap was exhausted.
Is there something I can do (other than change the application behavior) to
avoid this failure mode? I'm not the first to run into this, am I?
Thanks,
Dan
Re: 0.6.5 OOM during high read load
Posted by Dan Retzlaff <dr...@gmail.com>.
Beautiful, thanks.
On Sun, Mar 20, 2011 at 4:36 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> 0.7.1+ uses zero-copy reads in mmap'd mode so having 80k references to
> the same column is essentially just the reference overhead.
>
> On Fri, Mar 18, 2011 at 7:11 PM, Dan Retzlaff <dr...@gmail.com> wrote:
> > Dear experts, :)
> > Our application triggered an OOM error in Cassandra 0.6.5 by reading the
> > same 1.7MB column repeatedly (~80k reads). I analyzed the heap dump, and
> it
> > looks like the column value was queued 5400 times in an
> > OutboundTcpConnection destined for the Cassandra instance that received
> the
> > client request. Unfortunately, this intra-node connection goes across a
> > 100Mb data center interconnect, so it was only a matter of time before
> the
> > heap was exhausted.
> > Is there something I can do (other than change the application behavior)
> to
> > avoid this failure mode? I'm not the first to run into this, am I?
> > Thanks,
> > Dan
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: 0.6.5 OOM during high read load
Posted by Jonathan Ellis <jb...@gmail.com>.
0.7.1+ uses zero-copy reads in mmap'd mode so having 80k references to
the same column is essentially just the reference overhead.
On Fri, Mar 18, 2011 at 7:11 PM, Dan Retzlaff <dr...@gmail.com> wrote:
> Dear experts, :)
> Our application triggered an OOM error in Cassandra 0.6.5 by reading the
> same 1.7MB column repeatedly (~80k reads). I analyzed the heap dump, and it
> looks like the column value was queued 5400 times in an
> OutboundTcpConnection destined for the Cassandra instance that received the
> client request. Unfortunately, this intra-node connection goes across a
> 100Mb data center interconnect, so it was only a matter of time before the
> heap was exhausted.
> Is there something I can do (other than change the application behavior) to
> avoid this failure mode? I'm not the first to run into this, am I?
> Thanks,
> Dan
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com