You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by lebron james <le...@gmail.com> on 2011/07/24 14:08:28 UTC
Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Hi, Please help me with my problem. For better performance i turn off
compaction and run massive inserts, after database reach 37GB i stop massive
inserts and start compaction with "NodeTool compaction Keyspace CFamily".
after half hour of work cassandra fall with error "Out of memory" i give
1500M to JVM, all parameters in yaml file are default. testing OS ubuntu
11.04 and windows server 2008 dc edition. Thanks!
Re: Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Posted by Edward Capriolo <ed...@gmail.com>.
On Sunday, July 24, 2011, lebron james <le...@gmail.com> wrote:
> Hi, Please help me with my problem. For better performance i turn off
compaction and run massive inserts, after database reach 37GB i stop massive
inserts and start compaction with "NodeTool compaction Keyspace CFamily".
after half hour of work cassandra fall with error "Out of memory" i give
1500M to JVM, all parameters in yaml file are default. testing OS ubuntu
11.04 and windows server 2008 dc edition. Thanks!
>
Lebron good to see you have taken you talents from south beach to big data.
Though 1500mb is a lot of memory it is not a lot for cassandra. In
particular a burst of requests causes more memory allocation. Compaction is
intensive on the disk and memory system.
There are many things you can do to lower caches,optimize memtables, and
tune jvms. If you can spare more jvm memory that might help otherwise you
have a companion bug.
Re: Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Jul 27, 2011 at 7:40 AM, lebron james <le...@gmail.com> wrote:
> Why when i give jvm 1 GB heap size and try launch cassandra with 37GB
> database, cassandra start loading database, but after memory usage full,
> they fall with OutOfMemory Exception. Cassandra can`t work with low memory?
> or she critical need more RAM if database grows?
> here is link to download cassndra dist 0.8.2 with my 37GB database
> http://213.186.117.181/apache-cassandra-0.8.2-bin.zip
>
>
Cassandra's bloom filters must fit in memory. In addition each index file is
sampled (check the index_interval in cassandra.yaml)
# The Index Interval determines how large the sampling of row keys
# is for a given SSTable. The larger the sampling, the more effective
# the index is at the cost of space.
index_interval: 128
You can adjust index_interval to load large datasets with not-so-large
memory JVM, but 1GB heap for 37 GB database is not the best way to go.
How much ram does your system have?
Re: Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Posted by lebron james <le...@gmail.com>.
Why when i give jvm 1 GB heap size and try launch cassandra with 37GB
database, cassandra start loading database, but after memory usage full,
they fall with OutOfMemory Exception. Cassandra can`t work with low memory?
or she critical need more RAM if database grows?
here is link to download cassndra dist 0.8.2 with my 37GB database
http://213.186.117.181/apache-cassandra-0.8.2-bin.zip
Re: Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Posted by lebron james <le...@gmail.com>.
I zip last version of cassandra with yaml file and 37gb database, can
anybody download and do major compaction to check what is wrong? and why
cassandra fall in outofmemory exception. Tanks!
here is link to download
http://213.186.117.181/apache-cassandra-0.8.2-bin.zip
Re: Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Posted by lebron james <le...@gmail.com>.
I have only one CF with one UTF8 column and without indexes. in column
always 1 byte of data and keys is 16byte strings.
Re: Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Posted by aaron morton <aa...@thelastpickle.com>.
Have you tried some of the ideas about reducing the memory pressure ?
How many CF's + second indexes do you have?
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 26 Jul 2011, at 17:10, lebron james wrote:
> I have only 4GB on server, so i give jvm 3 GB of heap, but this dont help, cassandra still fall when i launch major compaction on 37 GB database.
>
>
>
Re: Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Posted by lebron james <le...@gmail.com>.
I have only 4GB on server, so i give jvm 3 GB of heap, but this dont help,
cassandra still fall when i launch major compaction on 37 GB database.
Re: Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Posted by aaron morton <aa...@thelastpickle.com>.
How much memory you need depends on a few things such as how many CF's you have, what your data is like, and what the usage patterns are like. There is no exact formula.
Generally…
* i would say 4GB of JVM heap is a good start
* key and row caches are set when the CF is created, see "help create column family" in cassandra-cli
* CF sizes are set using when running create column family, BUT under 0.8 there is a new automagical memory management see http://thelastpickle.com/2011/05/04/How-are-Memtables-measured/
* see the seting in_memory_compaction_limit_in_mb in cassandra.yaml for info on reducing the memory requirements for compaction.
The simplest things you can do is increase the JVM heap size to 3 or 4 GB.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 26 Jul 2011, at 00:35, lebron james wrote:
> >There are many things you can do to lower caches,optimize memtables, and tune jvms.
>
> Please tell what thins i can do to lower caches,optimize memtables, and tune jvms?
>
> >From experience with similar-sized data sets, 1.5GB may be too little. Recently I bumped our java HEAP limit from 3GB to 4GB to get ?>past an OOM doing a major compaction.
>
> In future i need database more than 10TB, so i need solve problem with ram, bacause i need use not more that 4 GB ram on 5TB database.
>
Re: Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Posted by lebron james <le...@gmail.com>.
>There are many things you can do to lower caches,optimize memtables, and
tune jvms.
Please tell what thins i can do to lower caches,optimize memtables, and
tune jvms?
>From experience with similar-sized data sets, 1.5GB may be too little.
Recently I bumped our java HEAP limit from 3GB to 4GB to get ?>past an OOM
doing a major compaction.
In future i need database more than 10TB, so i need solve problem with ram,
bacause i need use not more that 4 GB ram on 5TB database.
Re: Cassandra 0.7.8 and 0.8.1 fail when major compaction on 37GB database
Posted by Mina Naguib <mi...@bloomdigital.com>.
From experience with similar-sized data sets, 1.5GB may be too little. Recently I bumped our java HEAP limit from 3GB to 4GB to get past an OOM doing a major compaction.
Check "nodetool -h localhost info" while the compaction is running for a simple view into the memory state.
If you can, also hook in jconsole and you'll get a better view, over time, of how cassandra's memory usage trends, the effect of GC, and the pressure of various operations such as compactions.
On 2011-07-24, at 8:08 AM, lebron james wrote:
> Hi, Please help me with my problem. For better performance i turn off compaction and run massive inserts, after database reach 37GB i stop massive inserts and start compaction with "NodeTool compaction Keyspace CFamily". after half hour of work cassandra fall with error "Out of memory" i give 1500M to JVM, all parameters in yaml file are default. testing OS ubuntu 11.04 and windows server 2008 dc edition. Thanks!