You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Maxim Potekhin <po...@bnl.gov> on 2012/01/04 03:46:27 UTC
Strange OOM when doing "list" in CLI
I came back from Xmas vacation only to see that what always was an
innocuous procedure
in CLI now reliably results in OOM -- does anyone have ideas why?
It never happened before. Version of Cassandra is 0.8.8.
2956 java -ea
-javaagent:/home/cassandra/cassandra/bin/../lib/jamm-0.2.2.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 Xms8000M -Xmx8000M
-Xmn2000M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemakEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -X:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=199
[default@PANDA] list idxR;
Using default limit of 100
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:140)
at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:752)
at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734)
at
org.apache.cassandra.cli.CliClient.executeList(CliClient.java:1379)
at
org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:266)
at
org.apache.cassandra.cli.CliMain.processStatement(CliMain.java:217)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:345)
Re: Strange OOM when doing "list" in CLI
Posted by Maxim Potekhin <po...@bnl.gov>.
Ed,
thanks for a dose of common sense, I should have thunk about it.
In fact, I only have 2 columns in that one particular CF, but one of
these can get really fat (for a good reason). So the CLI just plain runs
out of memory when pulling the default 100 rows (with a little help from
various overheads). It didn't happen before because the recent additions
to the data were slightly fatter than in the beginning.
Thanks
Maxim
On 1/3/2012 10:27 PM, Edward Capriolo wrote:
> What you are probably running into is that list from the cli can bring
> all the columns of a key into memory. I have counters using composite
> keys and about 1k columns causes this to happen. We should have some
> paging support with list.
>
> On Tuesday, January 3, 2012, Maxim Potekhin <potekhin@bnl.gov
> <ma...@bnl.gov>> wrote:
> > I came back from Xmas vacation only to see that what always was an
> innocuous procedure
> > in CLI now reliably results in OOM -- does anyone have ideas why?
> >
> > It never happened before. Version of Cassandra is 0.8.8.
> >
> > 2956 java -ea
> -javaagent:/home/cassandra/cassandra/bin/../lib/jamm-0.2.2.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 Xms8000M
> -Xmx8000M -Xmn2000M -XX:+HeapDumpOnOutOfMemoryError -Xss128k
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemakEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -X:+PrintGCDetails
> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=199
> >
> >
> > [default@PANDA] list idxR;
> > Using default limit of 100
> > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> > at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:140)
> > at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> > at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> > at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> > at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> > at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> > at
> org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:752)
> > at
> org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734)
> > at
> org.apache.cassandra.cli.CliClient.executeList(CliClient.java:1379)
> > at
> org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:266)
> > at
> org.apache.cassandra.cli.CliMain.processStatement(CliMain.java:217)
> > at org.apache.cassandra.cli.CliMain.main(CliMain.java:345)
> >
> >
> >
Re: Strange OOM when doing "list" in CLI
Posted by Edward Capriolo <ed...@gmail.com>.
What you are probably running into is that list from the cli can bring all
the columns of a key into memory. I have counters using composite keys and
about 1k columns causes this to happen. We should have some paging support
with list.
On Tuesday, January 3, 2012, Maxim Potekhin <po...@bnl.gov> wrote:
> I came back from Xmas vacation only to see that what always was an
innocuous procedure
> in CLI now reliably results in OOM -- does anyone have ideas why?
>
> It never happened before. Version of Cassandra is 0.8.8.
>
> 2956 java -ea
-javaagent:/home/cassandra/cassandra/bin/../lib/jamm-0.2.2.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 Xms8000M -Xmx8000M
-Xmn2000M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemakEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -X:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=199
>
>
> [default@PANDA] list idxR;
> Using default limit of 100
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:140)
> at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:752)
> at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734)
> at
org.apache.cassandra.cli.CliClient.executeList(CliClient.java:1379)
> at
org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:266)
> at
org.apache.cassandra.cli.CliMain.processStatement(CliMain.java:217)
> at org.apache.cassandra.cli.CliMain.main(CliMain.java:345)
>
>
>