You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by "pavel@kreado.com pop" <pa...@kreado.com> on 2009/12/01 14:25:50 UTC
stress insert benchmark & cassandra hangup/crash
I did simple insert benchmark in java (see code below). It uses 5
threads to write records in one CF.
After some point cassandra hangup, then crashes. I use single node
cassandra installation with commit log on separate hdd.
Hardware:
AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
4 GB RAM,
2xHDD Samsung 1TB 32MB cache 7200RPM
Cassandra config file is standard with my own file paths:
<CommitLogDirectory>/media/
HDD_1000_2/data/cassandra/commitlog</CommitLogDirectory>
<DataFileDirectories>
<DataFileDirectory>/media/HDD_1000_1/data/cassandra/data</DataFileDirectory>
</DataFileDirectories>
<CalloutLocation>/media/HDD_1000_1/data/cassandra/callouts</CalloutLocation>
<StagingFileDirectory>/media/HDD_1000_1/data/cassandra/staging</StagingFileDirectory>
and Keyspace 'suggy', CF 'Pages'
At benchmark begin i see lower performance (i think this is because of JIT):
10000 time 12.375
10000 time 12.577
10000 time 12.782
10000 time 13.483
10000 time 13.475
20000 time 4.849
20000 time 4.924
20000 time 5.437
20000 time 4.966
20000 time 5.554
30000 time 6.22
30000 time 5.852
30000 time 5.893
30000 time 5.965
in some points performance slightly degrades (cassandra is
writing/sorting/combining memtables):
110000 time 3.943
120000 time 3.3
120000 time 3.143
120000 time 3.366
120000 time 3.437
120000 time 7.191
130000 time 7.288
130000 time 7.456
130000 time 6.931
130000 time 7.317
130000 time 2.985
140000 time 1.982
140000 time 3.422
140000 time 4.511
140000 time 4.17
140000 time 3.579
150000 time 3.807
150000 time 3.086
but at some point is performance degrades to unacceptable value:
350000 time 4.397
360000 time 2.944
360000 time 5.922
360000 time 2.668
350000 time 4.251
360000 time 4.163
370000 time 3.262
370000 time 17.045
370000 time 17.546
360000 time 17.433
370000 time 17.259
380000 time 17.286
380000 time 17.921
380000 time 17.557
370000 time 32.203
380000 time 32.111
390000 time 54.099
390000 time 98.446
390000 time 140.735
400000 time 179.168
390000 time 201.667
380000 time 202.058
400000 time 121.411
400000 time 79.004
390000 time 17.275
410000 time 18.713
400000 time 18.67
410000 time 97.205
410000 time 161.88
410000 time 167.947
420000 time 273.025
400000 time 460.794
At this point i don't see any hdd activity, and cassandra thread eat
100% of one cpu core.
Last log entries at INFO level:
INFO - Compacting
[org.apache.cassandra.io.SSTableReader(path='/media/HDD_1000_1/data/cassandra/data/suggy/Pages-1-Data.db'),org.apache.cassandra.io.SSTableReader(path='/media/HDD_1000_1/data/cassandra/data/suggy/Pages-2-Data.db'),org.apache.cassandra.io.SSTableReader(path='/media/HDD_1000_1/data/cassandra/data/suggy/Pages-3-Data.db'),org.apache.cassandra.io.SSTableReader(path='/media/HDD_1000_1/data/cassandra/data/suggy/Pages-4-Data.db')]
INFO - Pages has reached its threshold; switching in a fresh Memtable
INFO - Enqueuing flush of Memtable(Pages)@27641662
INFO - Sorting Memtable(Pages)@27641662
INFO - Pages has reached its threshold; switching in a fresh Memtable
INFO - Enqueuing flush of Memtable(Pages)@7518988
INFO - Sorting Memtable(Pages)@7518988
INFO - Completed flushing
/media/HDD_1000_1/data/cassandra/data/suggy/Pages-5-Data.db
INFO - Writing Memtable(Pages)@17877278
INFO - Pages has reached its threshold; switching in a fresh Memtable
INFO - Enqueuing flush of Memtable(Pages)@6467519
INFO - Sorting Memtable(Pages)@6467519
INFO - Pages has reached its threshold; switching in a fresh Memtable
INFO - Enqueuing flush of Memtable(Pages)@25574355
INFO - Sorting Memtable(Pages)@25574355
INFO - Completed flushing
/media/HDD_1000_1/data/cassandra/data/suggy/Pages-7-Data.db
INFO - Writing Memtable(Pages)@3968125
After long pause cassandra crashes (i use 1GB for java heap):
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid21132.hprof ...
Heap dump file created [1196864899 bytes in 51.159 secs]
I tried this testcase on three mashines with latest development version
of cassandra and with 0.4.2. And i got the same problems.
May be there is some problem in my benchmark?
My benchmark code:
import java.util.List;
import java.io.UnsupportedEncodingException;
import org.apache.thrift.transport.TTransport;
import org.apache.thrift.transport.TSocket;
import org.apache.thrift.protocol.TProtocol;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.TException;
import org.apache.cassandra.service.*;
public class bench implements Runnable
{
public static void main(String[] args)
throws InterruptedException, TException, TimedOutException,
InvalidRequestException, UnavailableException,
UnsupportedEncodingException, NotFoundException
{
bench runnable = new bench();
runnable.startseek=10000000;
Thread thread = new Thread(runnable);
thread.start();
bench runnable2 = new bench();
runnable2.startseek=20000000;
Thread thread2 = new Thread(runnable2);
thread2.start();
bench runnable3 = new bench();
runnable3.startseek=30000000;
Thread thread3 = new Thread(runnable3);
thread3.start();
bench runnable4 = new bench();
runnable4.startseek=40000000;
Thread thread4 = new Thread(runnable4);
thread4.start();
bench runnable5 = new bench();
runnable5.startseek=50000000;
Thread thread5 = new Thread(runnable5);
thread5.start();
thread.join();
thread2.join();
thread3.join();
thread4.join();
thread5.join();
}
public int startseek;
public void run() {
try {
TTransport tr = new TSocket("localhost", 9160);
TProtocol proto = new TBinaryProtocol(tr);
Cassandra.Client client = new Cassandra.Client(proto);
tr.open();
// insert data
long timestamp = System.currentTimeMillis();
ColumnPath cp=new ColumnPath("Pages", null,
"v".getBytes("UTF-8"));
int endseek=startseek+1000000;
int report_delay=10000;
int report_right=0;
for(int i=startseek;i<endseek;i++) {
if (report_right>=report_delay) {
report_right=0;
long new_timestamp=System.currentTimeMillis();
System.out.println(String.valueOf(i-startseek)+" time
"+String.valueOf((new_timestamp-timestamp)/1000.0));
timestamp=new_timestamp;
}
client.insert("suggy",
String.valueOf(i),
cp,
"================================================================================".getBytes("UTF-8"),
timestamp,
ConsistencyLevel.ZERO);
report_right++;
}
tr.close();
}
catch (TException e) {System.out.println(e);}
catch (TimedOutException e) {System.out.println(e);}
catch (InvalidRequestException e) {System.out.println(e);}
catch (UnavailableException e) {System.out.println(e);}
catch (UnsupportedEncodingException e) {System.out.println(e);}
}
}
PS: Sorry for my english
Re: stress insert benchmark & cassandra hangup/crash
Posted by Eric Evans <ee...@rackspace.com>.
On Tue, 2009-12-01 at 15:25 +0200, pavel@kreado.com pop wrote:
> I did simple insert benchmark in java (see code below). It uses 5
> threads to write records in one CF.
> After some point cassandra hangup, then crashes. I use single node
> cassandra installation with commit log on separate hdd.
>
> Hardware:
> AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
> 4 GB RAM,
> 2xHDD Samsung 1TB 32MB cache 7200RPM
[ ... ]
> After long pause cassandra crashes (i use 1GB for java heap):
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid21132.hprof ...
> Heap dump file created [1196864899 bytes in 51.159 secs]
>
> I tried this testcase on three mashines with latest development
> version
> of cassandra and with 0.4.2. And i got the same problems.
[ ... ]
> client.insert("suggy",
> String.valueOf(i),
> cp,
>
> "================================================================================".getBytes("UTF-8"),
> timestamp,
> ConsistencyLevel.ZERO);
With ConsistencyLevel.ZERO writes are completely asynchronous, so one
possibility is that you are simply pushing more writes than the node can
handle and it's getting further and further behind, eventually
exhausting the heap.
You should probably also read
http://wiki.apache.org/cassandra/MemtableThresholds if you haven't
already.
--
Eric Evans
eevans@rackspace.com