You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ronald Park <ro...@cbs.com> on 2010/05/17 21:44:00 UTC
nodetool causing OOM?
Hello,
We are getting our feet wet with Cassandra and have a test environment
set up to do some heavy data insertion. [Heavy is relative: we are
talking about 1M inserts in a 3 hours test.
Twice while running these tests, when we've tried to use 'nodetool'
about an hour or so into the test, to run a command like 'info' or
'tpstats' and the Cassandra node we attached to raised OOM error.
Now, the timing might just be coincidental the first time this happened,
but after the second time, we're a bit suspicious. :)
Could it be that, because we hadn't run nodetool until well into the
test run (like after 300-500K inserts)? Would running nodetool near the
start of the test instantiate some objects on the heap while there's
plenty of memory and avoid the problem?
Or is it just because we're using mostly default settings in
storage-conf.xml that aren't expected to work for tests of this size?
Thanks for any info,
Ron
Re: nodetool causing OOM?
Posted by Jonathan Ellis <jb...@gmail.com>.
Yeah, there's really not a whole lot we can do about these Thrift
problems other than get Avro ready as a replacement, which we are
doing. :(
On Mon, May 17, 2010 at 2:37 PM, Nahor <na...@gmail.com> wrote:
> On 2010-05-17 12:51, Brandon Williams wrote:
>>
>> On Mon, May 17, 2010 at 2:44 PM, Ronald Park <ronald.park@cbs.com
>> <ma...@cbs.com>> wrote:
>>
>> Hello,
>>
>> We are getting our feet wet with Cassandra and have a test
>> environment set up to do some heavy data insertion. [Heavy is
>> relative: we are talking about 1M inserts in a 3 hours test.
>>
>> Twice while running these tests, when we've tried to use
>> 'nodetool' about an hour or so into the test, to run a command
>> like 'info' or 'tpstats' and the Cassandra node we attached to
>> raised OOM error.
>>
>>
>> Does it only happen under load, or does it also OOM when idle? If the
>> latter, it sounds like you're connecting it to the thrift port instead of
>> the jmx port, which would trigger
>> https://issues.apache.org/jira/browse/THRIFT-601
>>
>
> Same bug but a different way to trigger it: mixing framed and non-framed
> transport.
> I had switched a node to use framed transport to look at Chiton but didn't
> change the nodetool configuration.
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: nodetool causing OOM?
Posted by Nahor <na...@gmail.com>.
On 2010-05-17 12:51, Brandon Williams wrote:
> On Mon, May 17, 2010 at 2:44 PM, Ronald Park <ronald.park@cbs.com
> <ma...@cbs.com>> wrote:
>
> Hello,
>
> We are getting our feet wet with Cassandra and have a test
> environment set up to do some heavy data insertion. [Heavy is
> relative: we are talking about 1M inserts in a 3 hours test.
>
> Twice while running these tests, when we've tried to use
> 'nodetool' about an hour or so into the test, to run a command
> like 'info' or 'tpstats' and the Cassandra node we attached to
> raised OOM error.
>
>
> Does it only happen under load, or does it also OOM when idle? If the
> latter, it sounds like you're connecting it to the thrift port instead
> of the jmx port, which would trigger
> https://issues.apache.org/jira/browse/THRIFT-601
>
Same bug but a different way to trigger it: mixing framed and non-framed
transport.
I had switched a node to use framed transport to look at Chiton but
didn't change the nodetool configuration.
Re: nodetool causing OOM?
Posted by Ronald Park <ro...@cbs.com>.
Brandon Williams wrote:
> On Mon, May 17, 2010 at 2:44 PM, Ronald Park <ronald.park@cbs.com
> <ma...@cbs.com>> wrote:
>
> Hello,
>
> We are getting our feet wet with Cassandra and have a test
> environment set up to do some heavy data insertion. [Heavy is
> relative: we are talking about 1M inserts in a 3 hours test.
>
> Twice while running these tests, when we've tried to use 'nodetool'
> about an hour or so into the test, to run a command like 'info' or
> 'tpstats' and the Cassandra node we attached to raised OOM error.
>
>
> Does it only happen under load, or does it also OOM when idle? If the
> latter, it sounds like you're connecting it to the thrift port instead
> of the jmx port, which would
> trigger https://issues.apache.org/jira/browse/THRIFT-601
>
> -Brandon
Ah ha. Yes, I am 99% sure this is what happened. I know I choose the
wrong port the first time I ran it; the other developer I'm working with
thinks he did so as well.
Ron
Re: nodetool causing OOM?
Posted by Brandon Williams <dr...@gmail.com>.
On Mon, May 17, 2010 at 2:44 PM, Ronald Park <ro...@cbs.com> wrote:
> Hello,
>
> We are getting our feet wet with Cassandra and have a test environment set
> up to do some heavy data insertion. [Heavy is relative: we are talking about
> 1M inserts in a 3 hours test.
>
> Twice while running these tests, when we've tried to use 'nodetool' about
> an hour or so into the test, to run a command like 'info' or 'tpstats' and
> the Cassandra node we attached to raised OOM error.
>
Does it only happen under load, or does it also OOM when idle? If the
latter, it sounds like you're connecting it to the thrift port instead of
the jmx port, which would trigger
https://issues.apache.org/jira/browse/THRIFT-601
-Brandon