You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "rkmr.em@gmail.com" <rk...@gmail.com> on 2009/07/13 06:16:13 UTC
cassandra slows down after inserts
hi
im using the default config from the svn trunk for everything: tables,
column, supercolumn, single node etc.
im inserting rows like this
for u in users:
client.insert('Table1', str(u.uid), 'Super1:attrs:key', str(u.key),
timestamp, False)
after about 100k rows, cassandra completely slows down taking for ever to
insert or get_slice_super. the commitlog is about 70MB in size.
when i stop and start cassandra the logs are replayed and it is fast again,
till it reaches the same point when it becomes really slow.
is this expected? is there something in the config that can be changed so
this doesnt happen?
thanks
<Tables>
<Table Name="Table1">
<!-- The fraction of keys per sstable whose locations we
keep in memory in "mostly LRU" order. (JUST the key
locations, NOT any column values.)
The amount of memory used by the default setting of
0.01 is comparable to the amount used by the internal
per-sstable key index. Consider increasing this is
fine if you have fewer, wider rows. Set to 0 to
disable entirely.
-->
<KeysCachedFraction>0.01</KeysCachedFraction>
<!-- if FlushPeriodInMinutes is configured and positive, it will
be
flushed to disk with that period whether it is dirty or
not.
This is intended for lightly-used columnfamilies so that
they
do not prevent commitlog segments from being purged. -->
<ColumnFamily ColumnSort="Name" Name="Standard1"
FlushPeriodInMinutes="60"/>
<ColumnFamily ColumnSort="Name" Name="Standard2"/>
<ColumnFamily ColumnSort="Time" Name="StandardByTime1"/>
<ColumnFamily ColumnType="Super" Name="Super1"/>
</Table>
</Tables>
Re: cassandra slows down after inserts
Posted by Jonathan Ellis <jb...@gmail.com>.
See the wiki page I linked.
On Mon, Jul 13, 2009 at 8:06 AM, rkmr.em@gmail.com<rk...@gmail.com> wrote:
>
> how do i find out if JVM is GCing?
>
> On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <sa...@gmail.com>
> wrote:
>>
>> What hardware are you running one?
>
> dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
> operating system is fedora core 9
>
>
>> How long does the slowdown last ?
>
> i stopped inserting data after slowdown starts, and it is still slow now
> after over 10 hours.
> however if i stop cassandra and start it, it becomes super fast immediately.
> till i insert another 100k or so rows when it becomes really slow again.
>
>
>>
>> There are a few reasons for temporary slowdowns ... perhaps the JVM
>> started GCing?
>
> how do i find out if this is the cause?
>
>
>>
>> Cassandra spends time cleaning up the on-disk SSTables
>> in a process called compaction. This could cause the client to observe
>> a slowdown.
>>
>> Things you could try --
>> Reduce the Memtable size in the config files. (If GCing was the problem)
>> Increasing the number of SSTables written before compaction kicks in.
>
> can you tell me what numbers i should use?
>
> thanks!
>
>
Re: cassandra slows down after inserts
Posted by Kevin Castiglione <ke...@gmail.com>.
Jonathan,
good to know this is normal. but the reason i sent this is , once i stop and
start cassandra it gets ready in few seconds and insertion and get_slices
are super fast like few milliseconds.
but after it starts to slow down even after 10 hours it is still slow. why
is this the case?
On Mon, Jul 13, 2009 at 6:14 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> Cassandra is replaying the transaction log and preloading SSTable
> indexes. This is normal.
>
> On Mon, Jul 13, 2009 at 8:10 AM, rkmr.em@gmail.com<rk...@gmail.com>
> wrote:
> > when i stop cassandra and start it again, this is what is printed. it
> takes
> > just a couple of seconds for this to run.
> > and after that it becomes really fast.
> >
> >
> > Listening for transport dt_socket at address: 8888
> > DEBUG - Loading settings from ./../conf/storage-conf.xml
> > DEBUG - adding Super1 as 0
> > DEBUG - adding Standard2 as 1
> > DEBUG - adding Standard1 as 2
> > DEBUG - adding StandardByTime1 as 3
> > DEBUG - adding LocationInfo as 4
> > DEBUG - adding HintsColumnFamily as 5
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-9-Data.db: 400 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-52-Data.db: 300 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-92-Data.db: 300 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db: 751 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db: 100 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db: 50 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db: 100 ms.
> > INFO - Compacting
> >
> [/home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db]
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db: 0 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db: 50 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db: 0 ms.
> > INFO - Replaying
> > /home/mark/local/var/cassandra/commitlog/CommitLog-1247454203796.log
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db : 73600
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db : 84224
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db : 94848
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db : 105472
> > DEBUG - Expected bloom filter size : 105472
> > INFO - Compacted to
> > /home/mark/local/var/cassandra/data/Table1-Super1-139-Data.db.
> 0/28831084
> > bytes for 104856/104860 keys read/written. Time: 8119ms.
> > INFO - Flushing Memtable(Super1)@552364977
> > DEBUG - Submitting Super1 for compaction
> > INFO - Completed flushing Memtable(Super1)@552364977
> > INFO - Flushing Memtable(Standard1)@1290243769
> > DEBUG - Submitting Standard1 for compaction
> > INFO - Completed flushing Memtable(Standard1)@1290243769
> > INFO - Compacting
> >
> [/home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db]
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db : 256
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db : 512
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db : 768
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db : 1024
> > DEBUG - Expected bloom filter size : 1024
> > INFO - Compacted to
> > /home/mark/local/var/cassandra/data/Table1-Standard1-3-Data.db. 0/210
> bytes
> > for 0/1 keys read/written. Time: 301ms.
> > DEBUG - Starting to listen on 127.0.0.1:7001
> > INFO - Cassandra starting up...
> >
> >
> >
> > On Mon, Jul 13, 2009 at 6:06 AM, rkmr.em@gmail.com <rk...@gmail.com>
> > wrote:
> >>
> >> how do i find out if JVM is GCing?
> >>
> >> On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <sa...@gmail.com>
> >> wrote:
> >>>
> >>> What hardware are you running one?
> >>
> >> dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
> >> operating system is fedora core 9
> >>
> >>
> >>> How long does the slowdown last ?
> >>
> >> i stopped inserting data after slowdown starts, and it is still slow now
> >> after over 10 hours.
> >> however if i stop cassandra and start it, it becomes super fast
> >> immediately. till i insert another 100k or so rows when it becomes
> really
> >> slow again.
> >>
> >>
> >>>
> >>> There are a few reasons for temporary slowdowns ... perhaps the JVM
> >>> started GCing?
> >>
> >> how do i find out if this is the cause?
> >>
> >>
> >>>
> >>> Cassandra spends time cleaning up the on-disk SSTables
> >>> in a process called compaction. This could cause the client to observe
> >>> a slowdown.
> >>>
> >>> Things you could try --
> >>> Reduce the Memtable size in the config files. (If GCing was the
> problem)
> >>> Increasing the number of SSTables written before compaction kicks in.
> >>
> >> can you tell me what numbers i should use?
> >>
> >> thanks!
> >>
> >
> >
>
Re: cassandra slows down after inserts
Posted by "rkmr.em@gmail.com" <rk...@gmail.com>.
Jonathan,
good to know this is normal. but the reason i sent this is , once i stop and
start cassandra it gets ready in few seconds and insertion and get_slices
are super fast like few milliseconds.
but after it starts to slow down even after 10 hours it is still slow. why
is this the case?
On Mon, Jul 13, 2009 at 6:14 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> Cassandra is replaying the transaction log and preloading SSTable
> indexes. This is normal.
>
> On Mon, Jul 13, 2009 at 8:10 AM, rkmr.em@gmail.com<rk...@gmail.com>
> wrote:
> > when i stop cassandra and start it again, this is what is printed. it
> takes
> > just a couple of seconds for this to run.
> > and after that it becomes really fast.
> >
> >
> > Listening for transport dt_socket at address: 8888
> > DEBUG - Loading settings from ./../conf/storage-conf.xml
> > DEBUG - adding Super1 as 0
> > DEBUG - adding Standard2 as 1
> > DEBUG - adding Standard1 as 2
> > DEBUG - adding StandardByTime1 as 3
> > DEBUG - adding LocationInfo as 4
> > DEBUG - adding HintsColumnFamily as 5
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-9-Data.db: 400 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-52-Data.db: 300 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-92-Data.db: 300 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db: 751 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db: 100 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db: 50 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db: 100 ms.
> > INFO - Compacting
> >
> [/home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db]
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db: 0 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db: 50 ms.
> > DEBUG - INDEX LOAD TIME for
> > /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db: 0 ms.
> > INFO - Replaying
> > /home/mark/local/var/cassandra/commitlog/CommitLog-1247454203796.log
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db : 73600
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db : 84224
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db : 94848
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db : 105472
> > DEBUG - Expected bloom filter size : 105472
> > INFO - Compacted to
> > /home/mark/local/var/cassandra/data/Table1-Super1-139-Data.db.
> 0/28831084
> > bytes for 104856/104860 keys read/written. Time: 8119ms.
> > INFO - Flushing Memtable(Super1)@552364977
> > DEBUG - Submitting Super1 for compaction
> > INFO - Completed flushing Memtable(Super1)@552364977
> > INFO - Flushing Memtable(Standard1)@1290243769
> > DEBUG - Submitting Standard1 for compaction
> > INFO - Completed flushing Memtable(Standard1)@1290243769
> > INFO - Compacting
> >
> [/home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db]
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db : 256
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db : 512
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db : 768
> > DEBUG - index size for bloom filter calc for file :
> > /home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db : 1024
> > DEBUG - Expected bloom filter size : 1024
> > INFO - Compacted to
> > /home/mark/local/var/cassandra/data/Table1-Standard1-3-Data.db. 0/210
> bytes
> > for 0/1 keys read/written. Time: 301ms.
> > DEBUG - Starting to listen on 127.0.0.1:7001
> > INFO - Cassandra starting up...
> >
> >
> >
> > On Mon, Jul 13, 2009 at 6:06 AM, rkmr.em@gmail.com <rk...@gmail.com>
> > wrote:
> >>
> >> how do i find out if JVM is GCing?
> >>
> >> On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <sa...@gmail.com>
> >> wrote:
> >>>
> >>> What hardware are you running one?
> >>
> >> dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
> >> operating system is fedora core 9
> >>
> >>
> >>> How long does the slowdown last ?
> >>
> >> i stopped inserting data after slowdown starts, and it is still slow now
> >> after over 10 hours.
> >> however if i stop cassandra and start it, it becomes super fast
> >> immediately. till i insert another 100k or so rows when it becomes
> really
> >> slow again.
> >>
> >>
> >>>
> >>> There are a few reasons for temporary slowdowns ... perhaps the JVM
> >>> started GCing?
> >>
> >> how do i find out if this is the cause?
> >>
> >>
> >>>
> >>> Cassandra spends time cleaning up the on-disk SSTables
> >>> in a process called compaction. This could cause the client to observe
> >>> a slowdown.
> >>>
> >>> Things you could try --
> >>> Reduce the Memtable size in the config files. (If GCing was the
> problem)
> >>> Increasing the number of SSTables written before compaction kicks in.
> >>
> >> can you tell me what numbers i should use?
> >>
> >> thanks!
> >>
> >
> >
>
Re: cassandra slows down after inserts
Posted by Jonathan Ellis <jb...@gmail.com>.
Cassandra is replaying the transaction log and preloading SSTable
indexes. This is normal.
On Mon, Jul 13, 2009 at 8:10 AM, rkmr.em@gmail.com<rk...@gmail.com> wrote:
> when i stop cassandra and start it again, this is what is printed. it takes
> just a couple of seconds for this to run.
> and after that it becomes really fast.
>
>
> Listening for transport dt_socket at address: 8888
> DEBUG - Loading settings from ./../conf/storage-conf.xml
> DEBUG - adding Super1 as 0
> DEBUG - adding Standard2 as 1
> DEBUG - adding Standard1 as 2
> DEBUG - adding StandardByTime1 as 3
> DEBUG - adding LocationInfo as 4
> DEBUG - adding HintsColumnFamily as 5
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-9-Data.db: 400 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-52-Data.db: 300 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-92-Data.db: 300 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db: 751 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db: 100 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db: 50 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db: 100 ms.
> INFO - Compacting
> [/home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db]
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db: 0 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db: 50 ms.
> DEBUG - INDEX LOAD TIME for
> /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db: 0 ms.
> INFO - Replaying
> /home/mark/local/var/cassandra/commitlog/CommitLog-1247454203796.log
> DEBUG - index size for bloom filter calc for file :
> /home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db : 73600
> DEBUG - index size for bloom filter calc for file :
> /home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db : 84224
> DEBUG - index size for bloom filter calc for file :
> /home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db : 94848
> DEBUG - index size for bloom filter calc for file :
> /home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db : 105472
> DEBUG - Expected bloom filter size : 105472
> INFO - Compacted to
> /home/mark/local/var/cassandra/data/Table1-Super1-139-Data.db. 0/28831084
> bytes for 104856/104860 keys read/written. Time: 8119ms.
> INFO - Flushing Memtable(Super1)@552364977
> DEBUG - Submitting Super1 for compaction
> INFO - Completed flushing Memtable(Super1)@552364977
> INFO - Flushing Memtable(Standard1)@1290243769
> DEBUG - Submitting Standard1 for compaction
> INFO - Completed flushing Memtable(Standard1)@1290243769
> INFO - Compacting
> [/home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db]
> DEBUG - index size for bloom filter calc for file :
> /home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db : 256
> DEBUG - index size for bloom filter calc for file :
> /home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db : 512
> DEBUG - index size for bloom filter calc for file :
> /home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db : 768
> DEBUG - index size for bloom filter calc for file :
> /home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db : 1024
> DEBUG - Expected bloom filter size : 1024
> INFO - Compacted to
> /home/mark/local/var/cassandra/data/Table1-Standard1-3-Data.db. 0/210 bytes
> for 0/1 keys read/written. Time: 301ms.
> DEBUG - Starting to listen on 127.0.0.1:7001
> INFO - Cassandra starting up...
>
>
>
> On Mon, Jul 13, 2009 at 6:06 AM, rkmr.em@gmail.com <rk...@gmail.com>
> wrote:
>>
>> how do i find out if JVM is GCing?
>>
>> On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <sa...@gmail.com>
>> wrote:
>>>
>>> What hardware are you running one?
>>
>> dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
>> operating system is fedora core 9
>>
>>
>>> How long does the slowdown last ?
>>
>> i stopped inserting data after slowdown starts, and it is still slow now
>> after over 10 hours.
>> however if i stop cassandra and start it, it becomes super fast
>> immediately. till i insert another 100k or so rows when it becomes really
>> slow again.
>>
>>
>>>
>>> There are a few reasons for temporary slowdowns ... perhaps the JVM
>>> started GCing?
>>
>> how do i find out if this is the cause?
>>
>>
>>>
>>> Cassandra spends time cleaning up the on-disk SSTables
>>> in a process called compaction. This could cause the client to observe
>>> a slowdown.
>>>
>>> Things you could try --
>>> Reduce the Memtable size in the config files. (If GCing was the problem)
>>> Increasing the number of SSTables written before compaction kicks in.
>>
>> can you tell me what numbers i should use?
>>
>> thanks!
>>
>
>
Re: cassandra slows down after inserts
Posted by "rkmr.em@gmail.com" <rk...@gmail.com>.
when i stop cassandra and start it again, this is what is printed. it takes
just a couple of seconds for this to run.
and after that it becomes really fast.
Listening for transport dt_socket at address: 8888
DEBUG - Loading settings from ./../conf/storage-conf.xml
DEBUG - adding Super1 as 0
DEBUG - adding Standard2 as 1
DEBUG - adding Standard1 as 2
DEBUG - adding StandardByTime1 as 3
DEBUG - adding LocationInfo as 4
DEBUG - adding HintsColumnFamily as 5
DEBUG - INDEX LOAD TIME for
/home/mark/local/var/cassandra/data/Table1-Super1-9-Data.db: 400 ms.
DEBUG - INDEX LOAD TIME for
/home/mark/local/var/cassandra/data/Table1-Super1-52-Data.db: 300 ms.
DEBUG - INDEX LOAD TIME for
/home/mark/local/var/cassandra/data/Table1-Super1-92-Data.db: 300 ms.
DEBUG - INDEX LOAD TIME for
/home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db: 751 ms.
DEBUG - INDEX LOAD TIME for
/home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db: 100 ms.
DEBUG - INDEX LOAD TIME for
/home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db: 50 ms.
DEBUG - INDEX LOAD TIME for
/home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db: 100 ms.
INFO - Compacting
[/home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db,/home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db]
DEBUG - INDEX LOAD TIME for
/home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db: 0 ms.
DEBUG - INDEX LOAD TIME for
/home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db: 50 ms.
DEBUG - INDEX LOAD TIME for
/home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db: 0 ms.
INFO - Replaying
/home/mark/local/var/cassandra/commitlog/CommitLog-1247454203796.log
DEBUG - index size for bloom filter calc for file :
/home/mark/local/var/cassandra/data/Table1-Super1-138-Data.db : 73600
DEBUG - index size for bloom filter calc for file :
/home/mark/local/var/cassandra/data/Table1-Super1-150-Data.db : 84224
DEBUG - index size for bloom filter calc for file :
/home/mark/local/var/cassandra/data/Table1-Super1-152-Data.db : 94848
DEBUG - index size for bloom filter calc for file :
/home/mark/local/var/cassandra/data/Table1-Super1-154-Data.db : 105472
DEBUG - Expected bloom filter size : 105472
INFO - Compacted to
/home/mark/local/var/cassandra/data/Table1-Super1-139-Data.db. 0/28831084
bytes for 104856/104860 keys read/written. Time: 8119ms.
INFO - Flushing Memtable(Super1)@552364977
DEBUG - Submitting Super1 for compaction
INFO - Completed flushing Memtable(Super1)@552364977
INFO - Flushing Memtable(Standard1)@1290243769
DEBUG - Submitting Standard1 for compaction
INFO - Completed flushing Memtable(Standard1)@1290243769
INFO - Compacting
[/home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db,/home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db]
DEBUG - index size for bloom filter calc for file :
/home/mark/local/var/cassandra/data/Table1-Standard1-2-Data.db : 256
DEBUG - index size for bloom filter calc for file :
/home/mark/local/var/cassandra/data/Table1-Standard1-4-Data.db : 512
DEBUG - index size for bloom filter calc for file :
/home/mark/local/var/cassandra/data/Table1-Standard1-6-Data.db : 768
DEBUG - index size for bloom filter calc for file :
/home/mark/local/var/cassandra/data/Table1-Standard1-8-Data.db : 1024
DEBUG - Expected bloom filter size : 1024
INFO - Compacted to
/home/mark/local/var/cassandra/data/Table1-Standard1-3-Data.db. 0/210 bytes
for 0/1 keys read/written. Time: 301ms.
DEBUG - Starting to listen on 127.0.0.1:7001
INFO - Cassandra starting up...
On Mon, Jul 13, 2009 at 6:06 AM, rkmr.em@gmail.com <rk...@gmail.com>wrote:
>
> how do i find out if JVM is GCing?
>
> On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <sa...@gmail.com>wrote:
>
>> What hardware are you running one?
>
>
> dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
> operating system is fedora core 9
>
>
> How long does the slowdown last ?
>
>
> i stopped inserting data after slowdown starts, and it is still slow now
> after over 10 hours.
> however if i stop cassandra and start it, it becomes super fast
> immediately. till i insert another 100k or so rows when it becomes really
> slow again.
>
>
>
>>
>> There are a few reasons for temporary slowdowns ... perhaps the JVM
>> started GCing?
>
> how do i find out if this is the cause?
>
>
>
>> Cassandra spends time cleaning up the on-disk SSTables
>> in a process called compaction. This could cause the client to observe
>> a slowdown.
>>
>> Things you could try --
>> Reduce the Memtable size in the config files. (If GCing was the problem)
>> Increasing the number of SSTables written before compaction kicks in.
>>
>
> can you tell me what numbers i should use?
>
> thanks!
>
>
Re: cassandra slows down after inserts
Posted by "rkmr.em@gmail.com" <rk...@gmail.com>.
how do i find out if JVM is GCing?
On Sun, Jul 12, 2009 at 10:37 PM, Sandeep Tata <sa...@gmail.com>wrote:
> What hardware are you running one?
dual quadcore intel xeon 2.0 ghz, 32GB ram, and hardware raid config
operating system is fedora core 9
How long does the slowdown last ?
i stopped inserting data after slowdown starts, and it is still slow now
after over 10 hours.
however if i stop cassandra and start it, it becomes super fast immediately.
till i insert another 100k or so rows when it becomes really slow again.
>
> There are a few reasons for temporary slowdowns ... perhaps the JVM
> started GCing?
how do i find out if this is the cause?
> Cassandra spends time cleaning up the on-disk SSTables
> in a process called compaction. This could cause the client to observe
> a slowdown.
>
> Things you could try --
> Reduce the Memtable size in the config files. (If GCing was the problem)
> Increasing the number of SSTables written before compaction kicks in.
>
can you tell me what numbers i should use?
thanks!
Re: cassandra slows down after inserts
Posted by Jonathan Ellis <jb...@gmail.com>.
decrease
On Mon, Jul 13, 2009 at 8:53 AM, rkmr.em@gmail.com<rk...@gmail.com> wrote:
> On Mon, Jul 13, 2009 at 6:03 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> On Mon, Jul 13, 2009 at 12:37 AM, Sandeep Tata<sa...@gmail.com>
>> wrote:
>> > What hardware are you running one? How long does the slowdown last ?
>> > There are a few reasons for temporary slowdowns ... perhaps the JVM
>> > started GCing?
>>
>> Every time someone has reported this symptom, that has been the problem.
>>
>> The object count tunable is the most direct way to ameliorate this.
>>
>> http://wiki.apache.org/cassandra/MemtableThresholds
>
> this is my current setting:
>
> <MemtableObjectCountInMillions>0.02</MemtableObjectCountInMillions>
>
> should i increase or decrease it?
>
Re: cassandra slows down after inserts
Posted by "rkmr.em@gmail.com" <rk...@gmail.com>.
On Mon, Jul 13, 2009 at 6:03 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Mon, Jul 13, 2009 at 12:37 AM, Sandeep Tata<sa...@gmail.com>
> wrote:
> > What hardware are you running one? How long does the slowdown last ?
> > There are a few reasons for temporary slowdowns ... perhaps the JVM
> > started GCing?
>
> Every time someone has reported this symptom, that has been the problem.
>
> The object count tunable is the most direct way to ameliorate this.
>
> http://wiki.apache.org/cassandra/MemtableThresholds
>
this is my current setting:
<MemtableObjectCountInMillions>0.02</MemtableObjectCountInMillions>
should i increase or decrease it?
Re: cassandra slows down after inserts
Posted by Jonathan Ellis <jb...@gmail.com>.
On Mon, Jul 13, 2009 at 12:37 AM, Sandeep Tata<sa...@gmail.com> wrote:
> What hardware are you running one? How long does the slowdown last ?
> There are a few reasons for temporary slowdowns ... perhaps the JVM
> started GCing?
Every time someone has reported this symptom, that has been the problem.
The object count tunable is the most direct way to ameliorate this.
http://wiki.apache.org/cassandra/MemtableThresholds
Re: cassandra slows down after inserts
Posted by Sandeep Tata <sa...@gmail.com>.
What hardware are you running one? How long does the slowdown last ?
There are a few reasons for temporary slowdowns ... perhaps the JVM
started GCing? Cassandra spends time cleaning up the on-disk SSTables
in a process called compaction. This could cause the client to observe
a slowdown.
Things you could try --
Reduce the Memtable size in the config files. (If GCing was the problem)
Increasing the number of SSTables written before compaction kicks in.
On Sun, Jul 12, 2009 at 9:16 PM, rkmr.em@gmail.com<rk...@gmail.com> wrote:
> hi
> im using the default config from the svn trunk for everything: tables,
> column, supercolumn, single node etc.
>
> im inserting rows like this
>
> for u in users:
> client.insert('Table1', str(u.uid), 'Super1:attrs:key', str(u.key),
> timestamp, False)
>
> after about 100k rows, cassandra completely slows down taking for ever to
> insert or get_slice_super. the commitlog is about 70MB in size.
> when i stop and start cassandra the logs are replayed and it is fast again,
> till it reaches the same point when it becomes really slow.
>
> is this expected? is there something in the config that can be changed so
> this doesnt happen?
>
> thanks
>
>
>
> <Tables>
> <Table Name="Table1">
> <!-- The fraction of keys per sstable whose locations we
> keep in memory in "mostly LRU" order. (JUST the key
> locations, NOT any column values.)
>
> The amount of memory used by the default setting of
> 0.01 is comparable to the amount used by the internal
> per-sstable key index. Consider increasing this is
> fine if you have fewer, wider rows. Set to 0 to
> disable entirely.
> -->
> <KeysCachedFraction>0.01</KeysCachedFraction>
> <!-- if FlushPeriodInMinutes is configured and positive, it will
> be
> flushed to disk with that period whether it is dirty or
> not.
> This is intended for lightly-used columnfamilies so that
> they
> do not prevent commitlog segments from being purged. -->
> <ColumnFamily ColumnSort="Name" Name="Standard1"
> FlushPeriodInMinutes="60"/>
> <ColumnFamily ColumnSort="Name" Name="Standard2"/>
> <ColumnFamily ColumnSort="Time" Name="StandardByTime1"/>
> <ColumnFamily ColumnType="Super" Name="Super1"/>
> </Table>
> </Tables>
>
>