You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Ran Tavory <ra...@gmail.com> on 2010/09/18 18:18:15 UTC
OOM when running hector's unit tests at 0.7.0
I started seeing OOM when running hector's unit-tests on 0.7.0. The OOM is
in cassandra's code so either there had been recent changes within cassandra
or it's just that we added new coverage which causes this OOM.
On my mac it's 100% repro but other hector devs have reported they don't get
this OOM. Moreover, it I place a Thread.sleep(5000) in one of the tests the
OOM goes away (but other errors are still present see more below)
To reproduce the error you may checkout hector from
http://github.com/rantav/hector, switch to the 0.7.0 branch and run:
$ MAVEN_OPTS="-Xmx1G -Xms256M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1" *mvn test -Dtest=CassandraClusterTest*
(I'm using the default memory settings from 0.7.0)
The test class source code is here
http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java
This test basically adds, removes, renames keyspaces etc.
Here's the OOM log:
18:07:04,197 INFO ColumnFamilyStore:734 - Enqueuing flush of
Memtable-Schema@72177013(3455 bytes, 3 operations)
18:07:04,487 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
Thread[FLUSH-WRITER-POOL:1,5,main]
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
heap space
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
at
org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
18:07:04,491 INFO Memtable:150 - Writing Memtable-Schema@72177013(3455
bytes, 3 operations)
18:07:04,812 INFO CompactionManager:323 - Compacted to
/Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-6-Data.db.
15,707 to 15,173 (~96% of original) bytes for 5 keys. Time: 860ms.
18:07:04,812 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
Thread[FLUSH-WRITER-POOL:1,5,main]
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
heap space
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
at
org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
Here the log after I place a sleep(5000) at this line
http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java#L88
The file-not-found error is even more suspicious...
18:04:28,293 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
Thread[CompactionExecutor:1,1,main]
java.util.concurrent.ExecutionException: java.io.FileNotFoundException:
/Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-7-Filter.db
(No such file or directory)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
at
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:604)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.io.FileNotFoundException:
/Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-7-Filter.db
(No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
at java.io.FileOutputStream.<init>(FileOutputStream.java:70)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:382)
at
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:144)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:314)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:107)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:88)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
$ java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-9M3165)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
Re: OOM when running hector's unit tests at 0.7.0
Posted by Ran Tavory <ra...@gmail.com>.
actually the tests are forked so the memory settings need to be set at the
pom.xml file and not by using maven_opts as I did. Ill experiment some more.
> An additional point - we just added the system_* methods and this test
> class does an add, rename then drop for both CF and keyspace.
>
> On 9/18/10, Ran Tavory <ra...@gmail.com> wrote:
>> I started seeing OOM when running hector's unit-tests on 0.7.0. The OOM
is
>> in cassandra's code so either there had been recent changes within
cassandra
>> or it's just that we added new coverage which causes this OOM.
>>
>> On my mac it's 100% repro but other hector devs have reported they don't
get
>> this OOM. Moreover, it I place a Thread.sleep(5000) in one of the tests
the
>> OOM goes away (but other errors are still present see more below)
>>
>> To reproduce the error you may checkout hector from
>> http://github.com/rantav/hector, switch to the 0.7.0 branch and run:
>>
>> $ MAVEN_OPTS="-Xmx1G -Xms256M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
>> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>> -XX:MaxTenuringThreshold=1" *mvn test -Dtest=CassandraClusterTest*
>>
>> (I'm using the default memory settings from 0.7.0)
>>
>> The test class source code is here
>>
http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java
>>
>> This test basically adds, removes, renames keyspaces etc.
>>
>> Here's the OOM log:
>>
>> 18:07:04,197 INFO ColumnFamilyStore:734 - Enqueuing flush of
>> Memtable-Schema@72177013(3455 bytes, 3 operations)
>> 18:07:04,487 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
>> Thread[FLUSH-WRITER-POOL:1,5,main]
>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
>> heap space
>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at
>>
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> at
>>
org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
>> at
>>
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
>> at
>>
org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
>> at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
>> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
>> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
>> at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> ... 2 more
>> 18:07:04,491 INFO Memtable:150 - Writing Memtable-Schema@72177013(3455
>> bytes, 3 operations)
>> 18:07:04,812 INFO CompactionManager:323 - Compacted to
>>
/Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-6-Data.db.
>> 15,707 to 15,173 (~96% of original) bytes for 5 keys. Time: 860ms.
>> 18:07:04,812 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
>> Thread[FLUSH-WRITER-POOL:1,5,main]
>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
>> heap space
>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at
>>
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> at
>>
org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
>> at
>>
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
>> at
>>
org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
>> at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
>> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
>> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
>> at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> ... 2 more
>>
>>
>> Here the log after I place a sleep(5000) at this line
>>
http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java#L88
>> The file-not-found error is even more suspicious...
>>
>> 18:04:28,293 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
>> Thread[CompactionExecutor:1,1,main]
>> java.util.concurrent.ExecutionException: java.io.FileNotFoundException:
>>
/Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-7-Filter.db
>> (No such file or directory)
>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at
>>
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
>> at
>>
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:604)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> Caused by: java.io.FileNotFoundException:
>>
/Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-7-Filter.db
>> (No such file or directory)
>> at java.io.FileOutputStream.open(Native Method)
>> at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
>> at java.io.FileOutputStream.<init>(FileOutputStream.java:70)
>> at
>>
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:382)
>> at
>>
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:144)
>> at
>>
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:314)
>> at
>>
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:107)
>> at
>>
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:88)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> ... 2 more
>>
>>
>> $ java -version
>> java version "1.6.0_20"
>> Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-9M3165)
>> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
>>
Re: OOM when running hector's unit tests at 0.7.0
Posted by Ran Tavory <ra...@gmail.com>.
FYI, when I increase the test memory to 1G to resemble the default setting
in cassandra the OOM doesn't happen, so that's a non-issue.
On Sat, Sep 18, 2010 at 9:02 PM, Ran Tavory <ra...@gmail.com> wrote:
> I haven't tried the python tests. What's also interesting is that if I
> run a single java test case then there's no oom. Only when I run the
> entire class then I get it. Also if I ads random sleeps then there is
> no oom but other more interesting errors surface.
>
> On Saturday, September 18, 2010, Nate McCall <na...@riptano.com> wrote:
> > Ran, have you tried to reproduce this using cassandra's python system
> > tests? Those have coverage of the system methods as well.
> >
> > On 9/18/10, Nate McCall <na...@riptano.com> wrote:
> >> An additional point - we just added the system_* methods and this test
> >> class does an add, rename then drop for both CF and keyspace.
> >>
> >> On 9/18/10, Ran Tavory <ra...@gmail.com> wrote:
> >>> I started seeing OOM when running hector's unit-tests on 0.7.0. The OOM
> >>> is
> >>> in cassandra's code so either there had been recent changes within
> >>> cassandra
> >>> or it's just that we added new coverage which causes this OOM.
> >>>
> >>> On my mac it's 100% repro but other hector devs have reported they
> don't
> >>> get
> >>> this OOM. Moreover, it I place a Thread.sleep(5000) in one of the tests
> >>> the
> >>> OOM goes away (but other errors are still present see more below)
> >>>
> >>> To reproduce the error you may checkout hector from
> >>> http://github.com/rantav/hector, switch to the 0.7.0 branch and run:
> >>>
> >>> $ MAVEN_OPTS="-Xmx1G -Xms256M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> >>> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> >>> -XX:MaxTenuringThreshold=1" *mvn test -Dtest=CassandraClusterTest*
> >>>
> >>> (I'm using the default memory settings from 0.7.0)
> >>>
> >>> The test class source code is here
> >>>
> http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java
> >>>
> >>> This test basically adds, removes, renames keyspaces etc.
> >>>
> >>> Here's the OOM log:
> >>>
> >>> 18:07:04,197 INFO ColumnFamilyStore:734 - Enqueuing flush of
> >>> Memtable-Schema@72177013(3455 bytes, 3 operations)
> >>> 18:07:04,487 ERROR AbstractCassandraDaemon:88 - Fatal exception in
> thread
> >>> Thread[FLUSH-WRITER-POOL:1,5,main]
> >>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
> Java
> >>> heap space
> >>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> >>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >>> at
> >>>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
> >>> at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> >>> at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>> at java.lang.Thread.run(Thread.java:637)
> >>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >>> at
> >>>
> org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
> >>> at
> >>>
> org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
> >>> at
> >>>
> org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
> >>> at
> >>> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
> >>> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
> >>> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
> >>> at
> >>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >>> at
> >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>> at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>> ... 2 more
> >>> 18:07:04,491 INFO Memtable:150 - Writing Memtable-Schema@72177013
> (3455
> >>> bytes, 3 operations)
> >>> 18:07:04,812 INFO CompactionManager:323 - Compacted to
> >>>
> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-6-Data.db.
> >>> 15,707 to 15,173 (~96% of original) bytes for 5 keys
>
> --
> /Ran
>
--
/Ran
Re: OOM when running hector's unit tests at 0.7.0
Posted by Ran Tavory <ra...@gmail.com>.
I haven't tried the python tests. What's also interesting is that if I
run a single java test case then there's no oom. Only when I run the
entire class then I get it. Also if I ads random sleeps then there is
no oom but other more interesting errors surface.
On Saturday, September 18, 2010, Nate McCall <na...@riptano.com> wrote:
> Ran, have you tried to reproduce this using cassandra's python system
> tests? Those have coverage of the system methods as well.
>
> On 9/18/10, Nate McCall <na...@riptano.com> wrote:
>> An additional point - we just added the system_* methods and this test
>> class does an add, rename then drop for both CF and keyspace.
>>
>> On 9/18/10, Ran Tavory <ra...@gmail.com> wrote:
>>> I started seeing OOM when running hector's unit-tests on 0.7.0. The OOM
>>> is
>>> in cassandra's code so either there had been recent changes within
>>> cassandra
>>> or it's just that we added new coverage which causes this OOM.
>>>
>>> On my mac it's 100% repro but other hector devs have reported they don't
>>> get
>>> this OOM. Moreover, it I place a Thread.sleep(5000) in one of the tests
>>> the
>>> OOM goes away (but other errors are still present see more below)
>>>
>>> To reproduce the error you may checkout hector from
>>> http://github.com/rantav/hector, switch to the 0.7.0 branch and run:
>>>
>>> $ MAVEN_OPTS="-Xmx1G -Xms256M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
>>> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>>> -XX:MaxTenuringThreshold=1" *mvn test -Dtest=CassandraClusterTest*
>>>
>>> (I'm using the default memory settings from 0.7.0)
>>>
>>> The test class source code is here
>>> http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java
>>>
>>> This test basically adds, removes, renames keyspaces etc.
>>>
>>> Here's the OOM log:
>>>
>>> 18:07:04,197 INFO ColumnFamilyStore:734 - Enqueuing flush of
>>> Memtable-Schema@72177013(3455 bytes, 3 operations)
>>> 18:07:04,487 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
>>> Thread[FLUSH-WRITER-POOL:1,5,main]
>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
>>> heap space
>>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>> at
>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:637)
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>> at
>>> org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
>>> at
>>> org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
>>> at
>>> org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
>>> at
>>> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
>>> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
>>> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
>>> at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> ... 2 more
>>> 18:07:04,491 INFO Memtable:150 - Writing Memtable-Schema@72177013(3455
>>> bytes, 3 operations)
>>> 18:07:04,812 INFO CompactionManager:323 - Compacted to
>>> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-6-Data.db.
>>> 15,707 to 15,173 (~96% of original) bytes for 5 keys
--
/Ran
Re: OOM when running hector's unit tests at 0.7.0
Posted by Nate McCall <na...@riptano.com>.
Ran, have you tried to reproduce this using cassandra's python system
tests? Those have coverage of the system methods as well.
On 9/18/10, Nate McCall <na...@riptano.com> wrote:
> An additional point - we just added the system_* methods and this test
> class does an add, rename then drop for both CF and keyspace.
>
> On 9/18/10, Ran Tavory <ra...@gmail.com> wrote:
>> I started seeing OOM when running hector's unit-tests on 0.7.0. The OOM
>> is
>> in cassandra's code so either there had been recent changes within
>> cassandra
>> or it's just that we added new coverage which causes this OOM.
>>
>> On my mac it's 100% repro but other hector devs have reported they don't
>> get
>> this OOM. Moreover, it I place a Thread.sleep(5000) in one of the tests
>> the
>> OOM goes away (but other errors are still present see more below)
>>
>> To reproduce the error you may checkout hector from
>> http://github.com/rantav/hector, switch to the 0.7.0 branch and run:
>>
>> $ MAVEN_OPTS="-Xmx1G -Xms256M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
>> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>> -XX:MaxTenuringThreshold=1" *mvn test -Dtest=CassandraClusterTest*
>>
>> (I'm using the default memory settings from 0.7.0)
>>
>> The test class source code is here
>> http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java
>>
>> This test basically adds, removes, renames keyspaces etc.
>>
>> Here's the OOM log:
>>
>> 18:07:04,197 INFO ColumnFamilyStore:734 - Enqueuing flush of
>> Memtable-Schema@72177013(3455 bytes, 3 operations)
>> 18:07:04,487 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
>> Thread[FLUSH-WRITER-POOL:1,5,main]
>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
>> heap space
>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at
>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
>> at
>> org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
>> at
>> org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
>> at
>> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
>> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
>> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> ... 2 more
>> 18:07:04,491 INFO Memtable:150 - Writing Memtable-Schema@72177013(3455
>> bytes, 3 operations)
>> 18:07:04,812 INFO CompactionManager:323 - Compacted to
>> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-6-Data.db.
>> 15,707 to 15,173 (~96% of original) bytes for 5 keys. Time: 860ms.
>> 18:07:04,812 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
>> Thread[FLUSH-WRITER-POOL:1,5,main]
>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
>> heap space
>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at
>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
>> at
>> org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
>> at
>> org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
>> at
>> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
>> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
>> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> ... 2 more
>>
>>
>> Here the log after I place a sleep(5000) at this line
>> http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java#L88
>> The file-not-found error is even more suspicious...
>>
>> 18:04:28,293 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
>> Thread[CompactionExecutor:1,1,main]
>> java.util.concurrent.ExecutionException: java.io.FileNotFoundException:
>> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-7-Filter.db
>> (No such file or directory)
>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at
>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
>> at
>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:604)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> Caused by: java.io.FileNotFoundException:
>> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-7-Filter.db
>> (No such file or directory)
>> at java.io.FileOutputStream.open(Native Method)
>> at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
>> at java.io.FileOutputStream.<init>(FileOutputStream.java:70)
>> at
>> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:382)
>> at
>> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:144)
>> at
>> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:314)
>> at
>> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:107)
>> at
>> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:88)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> ... 2 more
>>
>>
>> $ java -version
>> java version "1.6.0_20"
>> Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-9M3165)
>> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
>>
>
Re: OOM when running hector's unit tests at 0.7.0
Posted by Nate McCall <na...@riptano.com>.
An additional point - we just added the system_* methods and this test
class does an add, rename then drop for both CF and keyspace.
On 9/18/10, Ran Tavory <ra...@gmail.com> wrote:
> I started seeing OOM when running hector's unit-tests on 0.7.0. The OOM is
> in cassandra's code so either there had been recent changes within cassandra
> or it's just that we added new coverage which causes this OOM.
>
> On my mac it's 100% repro but other hector devs have reported they don't get
> this OOM. Moreover, it I place a Thread.sleep(5000) in one of the tests the
> OOM goes away (but other errors are still present see more below)
>
> To reproduce the error you may checkout hector from
> http://github.com/rantav/hector, switch to the 0.7.0 branch and run:
>
> $ MAVEN_OPTS="-Xmx1G -Xms256M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1" *mvn test -Dtest=CassandraClusterTest*
>
> (I'm using the default memory settings from 0.7.0)
>
> The test class source code is here
> http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java
>
> This test basically adds, removes, renames keyspaces etc.
>
> Here's the OOM log:
>
> 18:07:04,197 INFO ColumnFamilyStore:734 - Enqueuing flush of
> Memtable-Schema@72177013(3455 bytes, 3 operations)
> 18:07:04,487 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
> Thread[FLUSH-WRITER-POOL:1,5,main]
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
> at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
> at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ... 2 more
> 18:07:04,491 INFO Memtable:150 - Writing Memtable-Schema@72177013(3455
> bytes, 3 operations)
> 18:07:04,812 INFO CompactionManager:323 - Compacted to
> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-6-Data.db.
> 15,707 to 15,173 (~96% of original) bytes for 5 keys. Time: 860ms.
> 18:07:04,812 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
> Thread[FLUSH-WRITER-POOL:1,5,main]
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
> at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
> at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ... 2 more
>
>
> Here the log after I place a sleep(5000) at this line
> http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java#L88
> The file-not-found error is even more suspicious...
>
> 18:04:28,293 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread
> Thread[CompactionExecutor:1,1,main]
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException:
> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-7-Filter.db
> (No such file or directory)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
> at
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:604)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> Caused by: java.io.FileNotFoundException:
> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-7-Filter.db
> (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:70)
> at
> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:382)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:144)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:314)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:107)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:88)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ... 2 more
>
>
> $ java -version
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-9M3165)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
>