You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Lu Ming <xl...@live.com> on 2010/04/16 17:33:38 UTC
Cassandra 0.5.1 slow down after doing a lot of inserts
Hi:
We have build a storage system with 4 nodes in the cluster. We use the default configuration file,
every node have 2*E5504 CPU, 8G memory and 6*1T Sata disk. The cluster stores about 600G data.
We found that Cassandra 0.5.1 ALWAYS slow down after a lot of inserts. When it happens, CPU and Disk load
are very low. We use "tpstats" nodetools and find that more than 300,000 row-mutations is still pending and
the write speed is no more than 100~200 row-mutations per second.
It also happens when Cassandra starts and replays a large commitlog file. So we must cost several hours to
wait it finishs startup.
The Faq of Cassandra tells that it is caused by GC? any other explaining?
thanks a lot.
Re: Cassandra 0.5.1 slow down after doing a lot of inserts
Posted by Eric Evans <ee...@rackspace.com>.
On Fri, 2010-04-16 at 23:33 +0800, Lu Ming wrote:
> Hi:
> We have build a storage system with 4 nodes in the cluster. We use the default configuration file,
> every node have 2*E5504 CPU, 8G memory and 6*1T Sata disk. The cluster stores about 600G data.
>
> We found that Cassandra 0.5.1 ALWAYS slow down after a lot of inserts. When it happens, CPU and Disk load
> are very low. We use "tpstats" nodetools and find that more than 300,000 row-mutations is still pending and
> the write speed is no more than 100~200 row-mutations per second.
> It also happens when Cassandra starts and replays a large commitlog file. So we must cost several hours to
> wait it finishs startup.
>
> The Faq of Cassandra tells that it is caused by GC? any other explaining?
I assume you're referring to
http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
(which does indeed sound like what you are seeing). Following the advice
there should work.
Also, unless you have a very good reason for doing otherwise, I'd
recommend that you use 0.6.0, the latest stable release, instead of
0.5.1.
--
Eric Evans
eevans@rackspace.com
Re: Cassandra 0.5.1 slow down after doing a lot of inserts
Posted by Lu Ming <xl...@live.com>.
Cassandra often take >30 minutes to write about 300,000 row-mutations when
startup and replaying commitlog!
--------------------------------------------------
From: "Lu Ming" <xl...@live.com>
Sent: Friday, April 16, 2010 11:33 PM
To: <de...@cassandra.apache.org>
Subject: Cassandra 0.5.1 slow down after doing a lot of inserts
> Hi:
> We have build a storage system with 4 nodes in the cluster. We use the
> default configuration file,
> every node have 2*E5504 CPU, 8G memory and 6*1T Sata disk. The cluster
> stores about 600G data.
>
> We found that Cassandra 0.5.1 ALWAYS slow down after a lot of inserts.
> When it happens, CPU and Disk load
> are very low. We use "tpstats" nodetools and find that more than 300,000
> row-mutations is still pending and
> the write speed is no more than 100~200 row-mutations per second.
> It also happens when Cassandra starts and replays a large commitlog
> file. So we must cost several hours to
> wait it finishs startup.
>
> The Faq of Cassandra tells that it is caused by GC? any other
> explaining?
>
>
> thanks a lot.
Re: Cassandra 0.5.1 slow down after doing a lot of inserts
Posted by Jonathan Ellis <jb...@gmail.com>.
Moving to user list.
You should read
http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
2010/4/16 Lu Ming <xl...@live.com>:
> Hi:
> We have build a storage system with 4 nodes in the cluster. We use the default configuration file,
> every node have 2*E5504 CPU, 8G memory and 6*1T Sata disk. The cluster stores about 600G data.
>
> We found that Cassandra 0.5.1 ALWAYS slow down after a lot of inserts. When it happens, CPU and Disk load
> are very low. We use "tpstats" nodetools and find that more than 300,000 row-mutations is still pending and
> the write speed is no more than 100~200 row-mutations per second.
> It also happens when Cassandra starts and replays a large commitlog file. So we must cost several hours to
> wait it finishs startup.
>
> The Faq of Cassandra tells that it is caused by GC? any other explaining?
>
>
> thanks a lot.