You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Utku Can Topçu <ut...@topcu.gen.tr> on 2010/09/27 15:46:13 UTC
Best strategy for adding new nodes to the cluster
Hi All,
We're currently running a cassandra cluster with Replication Factor 3,
consisting of 4 nodes.
The current situation is:
- The nodes are all identical (AWS small instances)
- Data directory is in the partition (/mnt) which has 150G capacity and each
node has around 90 GB load, so 60 G free space per node is left.
So adding a new node to the cluster will seem to cause problems for us. I
think the node which will stream the data to the new bootstrapping node,
will not have enough disk space for anticompacting its data.
What should be the best practice for such scenarios?
Regards,
Utku
Re: Best strategy for adding new nodes to the cluster
Posted by Janne Jalkanen <ja...@ecyrd.com>.
On 28 Sep 2010, at 08:37, Michael Dürgner wrote:
>> What do you mean by "running live"? I am also planning to use cassandra on EC2 using small nodes. Small nodes have 1/4 cpu of the large ones, 1/4 cost, but I/O is more than 1/4 (amazon does not give explicit I/O numbers...), so I think 4 small instances should perform better than 1 large one (and the cost is the same), am I wrong?
>
> Based on results we saw and what you also find in different sources around the web, EC2 small instances perform worse than 1/4 regarding IO performance.
Ditto. My tests indicate that while the peak IO performance of small nodes can be ok (up to 1/2 of large), it degrades over time down to 1/6 or even less. It seems that Amazon dedicates sufficient bandwidth to small nodes in the beginning to ensure a smooth and quick boot, but then throttles down fairly aggressively within a few minutes. This seems to affect reads more than writes, though.
Note also that large instances have over 4x the memory (1.7 GB => 7.5 GB), and that makes a world of difference (you can have larger caches, for example). You don't really want to start swapping on the small instances.
(However, small instances are awesome for doing testing and learning how to manage a cluster.)
/Janne
Re: Best strategy for adding new nodes to the cluster
Posted by Michael Dürgner <mi...@duergner.de>.
Sent from my iPhone
On 27.09.2010, at 19:30, Marc Canaleta <mc...@gmail.com> wrote:
> What do you mean by "running live"? I am also planning to use cassandra on EC2 using small nodes. Small nodes have 1/4 cpu of the large ones, 1/4 cost, but I/O is more than 1/4 (amazon does not give explicit I/O numbers...), so I think 4 small instances should perform better than 1 large one (and the cost is the same), am I wrong?
Based on results we saw and what you also find in different sources around the web, EC2 small instances perform worse than 1/4 regarding IO performance.
> El 27 de septiembre de 2010 18:09:14 UTC+2, Jonathan Ellis <jb...@gmail.com> escribió:
> I strongly recommend not running live on Small nodes. So in your case
> I would recommend starting up Large instances with raid0'd disks, shut
> down cassandra on the Small ones, rsync to the Large, and start up on
> Large.
>
> On Mon, Sep 27, 2010 at 6:46 AM, Utku Can Topçu <ut...@topcu.gen.tr> wrote:
> > Hi All,
> >
> > We're currently running a cassandra cluster with Replication Factor 3,
> > consisting of 4 nodes.
> >
> > The current situation is:
> >
> > - The nodes are all identical (AWS small instances)
> > - Data directory is in the partition (/mnt) which has 150G capacity and each
> > node has around 90 GB load, so 60 G free space per node is left.
> >
> > So adding a new node to the cluster will seem to cause problems for us. I
> > think the node which will stream the data to the new bootstrapping node,
> > will not have enough disk space for anticompacting its data.
> >
> > What should be the best practice for such scenarios?
> >
> > Regards,
> >
> > Utku
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: Best strategy for adding new nodes to the cluster
Posted by Peter Schuller <pe...@infidyne.com>.
> What do you mean by "running live"? I am also planning to use cassandra on
I believe live as in "in production".
> EC2 using small nodes. Small nodes have 1/4 cpu of the large ones, 1/4 cost,
> but I/O is more than 1/4 (amazon does not give explicit I/O numbers...), so
> I think 4 small instances should perform better than 1 large one (and the
> cost is the same), am I wrong?
Without making any claims with respect to Jonathan's reasons for
saying so, I'd be hesitant for reasons including:
* 32 bit -> no mmap()
* Even assuming a fully dedicated CPU for that instance (which won't
be true), things like background compaction, memtable flushing and
concurrent GC would presumably tend to have larger impact (in
relative terms) on real traffc than on multi-CPU setups.
* I'd be generally skeptical about the potential variation in actual
available CPU and whether the variation will vary across instance
types. It would depend, presumably, quite a bit on luck and how Amazon
overcommits/allocates hosts for instances, but my very limited and
very anecdotal experience is that small instances seem to be more
over-committed (but this could be entirely wrong; don't take my word
for it - anyone else?).
On the other hand, if the only concern is I/O bandwidth rather than
CPU use maybe the situation is different. (Does anyone have numbers on
variation in available disk bandwidth over time for EC2 instances?)
--
/ Peter Schuller
Re: Best strategy for adding new nodes to the cluster
Posted by Marc Canaleta <mc...@gmail.com>.
What do you mean by "running live"? I am also planning to use cassandra on
EC2 using small nodes. Small nodes have 1/4 cpu of the large ones, 1/4 cost,
but I/O is more than 1/4 (amazon does not give explicit I/O numbers...), so
I think 4 small instances should perform better than 1 large one (and the
cost is the same), am I wrong?
El 27 de septiembre de 2010 18:09:14 UTC+2, Jonathan Ellis <
jbellis@gmail.com> escribió:
> I strongly recommend not running live on Small nodes. So in your case
> I would recommend starting up Large instances with raid0'd disks, shut
> down cassandra on the Small ones, rsync to the Large, and start up on
> Large.
>
> On Mon, Sep 27, 2010 at 6:46 AM, Utku Can Topçu <ut...@topcu.gen.tr> wrote:
> > Hi All,
> >
> > We're currently running a cassandra cluster with Replication Factor 3,
> > consisting of 4 nodes.
> >
> > The current situation is:
> >
> > - The nodes are all identical (AWS small instances)
> > - Data directory is in the partition (/mnt) which has 150G capacity and
> each
> > node has around 90 GB load, so 60 G free space per node is left.
> >
> > So adding a new node to the cluster will seem to cause problems for us. I
> > think the node which will stream the data to the new bootstrapping node,
> > will not have enough disk space for anticompacting its data.
> >
> > What should be the best practice for such scenarios?
> >
> > Regards,
> >
> > Utku
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: Best strategy for adding new nodes to the cluster
Posted by Jonathan Ellis <jb...@gmail.com>.
I strongly recommend not running live on Small nodes. So in your case
I would recommend starting up Large instances with raid0'd disks, shut
down cassandra on the Small ones, rsync to the Large, and start up on
Large.
On Mon, Sep 27, 2010 at 6:46 AM, Utku Can Topçu <ut...@topcu.gen.tr> wrote:
> Hi All,
>
> We're currently running a cassandra cluster with Replication Factor 3,
> consisting of 4 nodes.
>
> The current situation is:
>
> - The nodes are all identical (AWS small instances)
> - Data directory is in the partition (/mnt) which has 150G capacity and each
> node has around 90 GB load, so 60 G free space per node is left.
>
> So adding a new node to the cluster will seem to cause problems for us. I
> think the node which will stream the data to the new bootstrapping node,
> will not have enough disk space for anticompacting its data.
>
> What should be the best practice for such scenarios?
>
> Regards,
>
> Utku
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com