You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Henry Luo <hl...@choicestream.com> on 2010/10/14 22:23:35 UTC
Hundreds compaction a day, is it normal?
We have a five node cluster, using replication factor of 3. The application is only sending write requests at this point - we'd like to gain some operation experience with it first before start read from it.
We are seeing over a hundred compaction activities on each server, some of them are for HintsColumnFamily.
Each machine has 32 GB memory, two disk arrays, one with raid 0 for commit log, one with raid 5 for data. We are using version 0.6.1, and pretty much the out of the box storage.xml.
Is this normal? Where should we look for tuning?
Here is the ring info
Address Status Load Range Ring
103348149328693428942388257816272166328
10.100.10.68 Up 136.41 GB 62116456964768051843784433654721163092 |<--|
10.100.10.64 Up 136.31 GB 82105179051854269619799333828977372565 | ^
10.100.10.66 Up 152.77 GB 92197953251627500070365755299174650936 v |
10.100.10.72 Up 71.38 GB 102264937228017528105060257264614100661 | ^
10.100.10.76 Up 24.8 GB 103348149328693428942388257816272166328 |-->|
Thanks.
Henry
________________________________
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers.
Re: Hundreds compaction a day, is it normal?
Posted by Jonathan Ellis <jb...@gmail.com>.
On Thu, Oct 14, 2010 at 7:36 PM, Henry Luo <hl...@choicestream.com> wrote:
> Thanks for the advice. Follow up questions:
>
> a) is 0.6.6 compactable with 0.6.1?
Yes, you can upgrade one node at a time and it will particpate w/ the
0.6.1 nodes until they are done too. Just restart w/ 0.6.6, no data
migration or commitlog drain needed. No api changes were made.
> Do we need to change the config?
You should look at the changes because some defaults were improved. I
don't think any are mandatory but I wouldn't swear to it.
> b) Should we wait for 0.7? If so, same questions above.
0.7 can read 0.6 data files but is not a drop-in replacement. See
"Upgrading" in https://svn.apache.org/repos/asf/cassandra/trunk/NEWS.txt
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
RE: Hundreds compaction a day, is it normal?
Posted by Henry Luo <hl...@choicestream.com>.
Thanks for the advice. Follow up questions:
a) is 0.6.6 compactable with 0.6.1? Do we need to change the config? How about the data in the current system?
b) Should we wait for 0.7? If so, same questions above.
Thanks.
Henry
-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com]
Sent: Thursday, October 14, 2010 4:33 PM
To: user
Subject: Re: Hundreds compaction a day, is it normal?
a) 0.6.1 is ancient, upgrade to 0.6.6 (see
http://www.riptano.com/blog/whats-new-cassandra-066 for links to all
the improvements since 0.6.1 -- the links to older versions are at the
bottom)
b) increase the memtable flush thresholds to reduce the need for
compaction (8x the defaults is a decent starting point, assuming you
are giving a reasonable amount of your 32GB to the JVM for heap)
c) set compaction priority lower
(http://www.riptano.com/blog/cassandra-annotated-changelog-063) to
make compaction hurt less
d) enable the dynamic snitch
(http://www.riptano.com/blog/whats-new-cassandra-065) to route around
nodes that are slow b/c of compaction
On Thu, Oct 14, 2010 at 3:23 PM, Henry Luo <hl...@choicestream.com> wrote:
> We have a five node cluster, using replication factor of 3. The application
> is only sending write requests at this point - we'd like to gain some
> operation experience with it first before start read from it.
>
>
>
> We are seeing over a hundred compaction activities on each server, some of
> them are for HintsColumnFamily.
>
>
>
> Each machine has 32 GB memory, two disk arrays, one with raid 0 for commit
> log, one with raid 5 for data. We are using version 0.6.1, and pretty much
> the out of the box storage.xml.
>
>
>
> Is this normal? Where should we look for tuning?
>
>
>
> Here is the ring info
>
>
>
> Address Status Load
> Range Ring
>
>
> 103348149328693428942388257816272166328
>
> 10.100.10.68 Up 136.41 GB
> 62116456964768051843784433654721163092 |<--|
>
> 10.100.10.64 Up 136.31 GB
> 82105179051854269619799333828977372565 | ^
>
> 10.100.10.66 Up 152.77 GB
> 92197953251627500070365755299174650936 v |
>
> 10.100.10.72 Up 71.38 GB
> 102264937228017528105060257264614100661 | ^
>
> 10.100.10.76 Up 24.8 GB
> 103348149328693428942388257816272166328 |-->|
>
>
>
> Thanks.
>
> Henry
>
>
>
> ________________________________
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential, proprietary, and/or
> privileged material. Any review, retransmission, dissemination or other use
> of, or taking of any action in reliance upon this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from all
> computers.
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers.
Re: Hundreds compaction a day, is it normal?
Posted by Jonathan Ellis <jb...@gmail.com>.
a) 0.6.1 is ancient, upgrade to 0.6.6 (see
http://www.riptano.com/blog/whats-new-cassandra-066 for links to all
the improvements since 0.6.1 -- the links to older versions are at the
bottom)
b) increase the memtable flush thresholds to reduce the need for
compaction (8x the defaults is a decent starting point, assuming you
are giving a reasonable amount of your 32GB to the JVM for heap)
c) set compaction priority lower
(http://www.riptano.com/blog/cassandra-annotated-changelog-063) to
make compaction hurt less
d) enable the dynamic snitch
(http://www.riptano.com/blog/whats-new-cassandra-065) to route around
nodes that are slow b/c of compaction
On Thu, Oct 14, 2010 at 3:23 PM, Henry Luo <hl...@choicestream.com> wrote:
> We have a five node cluster, using replication factor of 3. The application
> is only sending write requests at this point – we’d like to gain some
> operation experience with it first before start read from it.
>
>
>
> We are seeing over a hundred compaction activities on each server, some of
> them are for HintsColumnFamily.
>
>
>
> Each machine has 32 GB memory, two disk arrays, one with raid 0 for commit
> log, one with raid 5 for data. We are using version 0.6.1, and pretty much
> the out of the box storage.xml.
>
>
>
> Is this normal? Where should we look for tuning?
>
>
>
> Here is the ring info
>
>
>
> Address Status Load
> Range Ring
>
>
> 103348149328693428942388257816272166328
>
> 10.100.10.68 Up 136.41 GB
> 62116456964768051843784433654721163092 |<--|
>
> 10.100.10.64 Up 136.31 GB
> 82105179051854269619799333828977372565 | ^
>
> 10.100.10.66 Up 152.77 GB
> 92197953251627500070365755299174650936 v |
>
> 10.100.10.72 Up 71.38 GB
> 102264937228017528105060257264614100661 | ^
>
> 10.100.10.76 Up 24.8 GB
> 103348149328693428942388257816272166328 |-->|
>
>
>
> Thanks.
>
> Henry
>
>
>
> ________________________________
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential, proprietary, and/or
> privileged material. Any review, retransmission, dissemination or other use
> of, or taking of any action in reliance upon this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from all
> computers.
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com