You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by rash aroskar <ra...@gmail.com> on 2013/08/02 21:39:42 UTC
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Thanks for helpful responses. The upgrade from 0.8 to 1.2 is not direct, we
have setup test cluster where we did upgrade from 0.8 to 1.1 and then 1.2.
Also we will do a whole different cluster with 1.2, the 0.8 cluster will
not be upgraded. But the data will be moved from 0.8 cluster to 1.2
cluster.
The reason for me looking at virtual nodes is because of terrible
experiences we had with 0.8 repairs and as per documentation (an logically)
the virtual nodes seems like it will help repairs being smoother. Is this
true? Also how to get the right number of virtual nodes? David suggested 64
vnodes for 20 machines. Is there a formula or a thought process to be
followed to get this number right?
On Mon, Jul 29, 2013 at 4:15 AM, aaron morton <aa...@thelastpickle.com>wrote:
> I would *strongly* recommend against upgrading from 0.8 directly to 1.2.
> Skipping a major version is generally not recommended, skipped 3 would seem
> like carelessness.
>
> I second Romain, do the upgrade and make sure the health is good first.
>
> +1 but I would also recommend deciding if you actually need to use virtual
> nodes. The shuffle process can take a long time and people have had mixed
> experiences with it.
>
> If you wanted to move to 1.2 and get vNodes I would consider spinning up a
> new cluster and bulk loading into it. You could do an initial load and then
> to delta loads using snapshots, there would however be a period of stale
> data in the new cluster until the last delta snapshot is loaded.
>
> Cheers
>
> -----------------
> Aaron Morton
> Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 27/07/2013, at 3:36 AM, David McNelis <dm...@gmail.com> wrote:
>
> I second Romain, do the upgrade and make sure the health is good first.
>
> If you have or plan to have a large number of nodes, you might consider
> using fewer than 256 as your initial vnodes amount. I think that number is
> inflated from reasonable in the docs, as we've had some people talk about
> potential performance degradation if you have a large number of nodes and a
> very high number of vnodes, if I had it to do over again, I'd have done 64
> vnodes as my default (across 20 nodes).
>
> Another thing to be very cognizant of before shuffle is disk space. You
> *must* have less than 50% used in order to do the shuffle successfully
> because no data is removed (cleaned) from a node during the shuffle process
> and the shuffle process essentially doubles the amount of data until you're
> able to run a clean.
>
>
> On Fri, Jul 26, 2013 at 11:25 AM, Romain HARDOUIN <
> romain.hardouin@urssaf.fr> wrote:
>
>> Vnodes are a great feature. More nodes are involved during operations
>> such as bootstrap, decommission, etc.
>> DataStax documentation is definitely a must read.
>> That said, If I were you, I'd wait somewhat before to shuffle the ring.
>> I'd focus on cluster upgrade and monitoring the nodes. (number of files
>> handles, memory usage, latency, etc).
>> Upgrading from 0.8 to 1.2 can be tricky, there are so many changes since
>> then. Be careful about compaction strategies you choose and double check
>> the options.
>>
>> Regards,
>> Romain
>>
>> rash aroskar <ra...@gmail.com> a écrit sur 25/07/2013 23:25:11 :
>>
>> > De : rash aroskar <ra...@gmail.com>
>> > A : user@cassandra.apache.org,
>> > Date : 25/07/2013 23:25
>> > Objet : cassandra 1.2.5- virtual nodes (num_token) pros/cons?
>> >
>> > Hi,
>> > I am upgrading my cassandra cluster from 0.8 to 1.2.5.
>> > In cassandra 1.2.5 the 'num_token' attribute confuses me.
>> > I understand that it distributes multiple tokens per node but I am
>> > not clear how that is helpful for performance or load balancing. Can
>> > anyone elaborate? has anyone used this feature and knows its
>> > advantages/disadvantages?
>> >
>> > Thanks,
>> > Rashmi
>>
>
>
>
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Posted by Aaron Morton <aa...@thelastpickle.com>.
> Repair and bootstrap will be limited by the node doing repair or bootstrap, since it has to do the same amount of work whatever num_tokens is.
That's what I was thinking.
I normally assume repair has very little data to stream, and most of the time is taken creating the merkle trees. In that case the node repair was started on still has to compact / hash all it's data, however hashing the replicas would go faster is it's done on more machines.
Cheers
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 13/08/2013, at 11:13 PM, Richard Low <ri...@wentnet.com> wrote:
> On 13 August 2013 10:15, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>
> Streaming from all the physical nodes in the cluster should make repair faster, for the same reason it makes bootstrap faster. Shouldn't it ?
>
> Virtual nodes doesn't speed up either very much. Repair and bootstrap will be limited by the node doing repair or bootstrap, since it has to do the same amount of work whatever num_tokens is. It places a more even load across the rest of the cluster though, since it will repair with or bootstrap from all nodes in the cluster. So the overall time will in most cases be about the same.
>
> The real speedup from vnodes comes when running removenode, when the streaming happens in parallel across all nodes.
>
> Richard.
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Posted by Richard Low <ri...@wentnet.com>.
On 13 August 2013 10:15, Alain RODRIGUEZ <ar...@gmail.com> wrote:
Streaming from all the physical nodes in the cluster should make repair
> faster, for the same reason it makes bootstrap faster. Shouldn't it ?
>
Virtual nodes doesn't speed up either very much. Repair and bootstrap will
be limited by the node doing repair or bootstrap, since it has to do the
same amount of work whatever num_tokens is. It places a more even load
across the rest of the cluster though, since it will repair with or
bootstrap from all nodes in the cluster. So the overall time will in most
cases be about the same.
The real speedup from vnodes comes when running removenode, when the
streaming happens in parallel across all nodes.
Richard.
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
@Aaron
Streaming from all the physical nodes in the cluster should make repair
faster, for the same reason it makes bootstrap faster. Shouldn't it ?
2013/8/12 Aaron Morton <aa...@thelastpickle.com>
> Aaron - I read about the virtual nodes at
> http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2
>
>
> Thanks, I did not see anything in there about making repair smoother /
> faster.
>
> Cheers
> A
> -----------------
> Aaron Morton
> Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 10/08/2013, at 11:39 AM, rash aroskar <ra...@gmail.com> wrote:
>
> Aaron - I read about the virtual nodes at
> http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2
>
>
>
>
>
>
>
>
> On Tue, Aug 6, 2013 at 4:49 AM, Richard Low <ri...@wentnet.com> wrote:
>
>> On 6 August 2013 08:40, Aaron Morton <aa...@thelastpickle.com> wrote:
>>
>>> The reason for me looking at virtual nodes is because of terrible
>>> experiences we had with 0.8 repairs and as per documentation (an logically)
>>> the virtual nodes seems like it will help repairs being smoother. Is this
>>> true?
>>>
>>> I've not thought too much about how they help repair run smoother, what
>>> was the documentation you read ?
>>>
>>
>> There might be a slight improvement but I haven't observed any. The
>> difference might be that, because every node shares replicas with every
>> other (with high probability), a single repair operation does the same work
>> on the node it was called on, but the rest is spread out over the cluster,
>> rather than just the RF nodes either side of the repairing node. This
>> means the post-repair compaction work will take less time and the length of
>> time a node is loaded for during repair is less.
>>
>> However, the other benefits of vnodes are likely to be much more useful.
>>
>> Richard.
>>
>
>
>
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Posted by Aaron Morton <aa...@thelastpickle.com>.
> Aaron - I read about the virtual nodes at http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2
Thanks, I did not see anything in there about making repair smoother / faster.
Cheers
A
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 10/08/2013, at 11:39 AM, rash aroskar <ra...@gmail.com> wrote:
> Aaron - I read about the virtual nodes at http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2
>
>
>
>
>
>
>
>
> On Tue, Aug 6, 2013 at 4:49 AM, Richard Low <ri...@wentnet.com> wrote:
> On 6 August 2013 08:40, Aaron Morton <aa...@thelastpickle.com> wrote:
>> The reason for me looking at virtual nodes is because of terrible experiences we had with 0.8 repairs and as per documentation (an logically) the virtual nodes seems like it will help repairs being smoother. Is this true?
> I've not thought too much about how they help repair run smoother, what was the documentation you read ?
>
> There might be a slight improvement but I haven't observed any. The difference might be that, because every node shares replicas with every other (with high probability), a single repair operation does the same work on the node it was called on, but the rest is spread out over the cluster, rather than just the RF nodes either side of the repairing node. This means the post-repair compaction work will take less time and the length of time a node is loaded for during repair is less.
>
> However, the other benefits of vnodes are likely to be much more useful.
>
> Richard.
>
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Posted by rash aroskar <ra...@gmail.com>.
Aaron - I read about the virtual nodes at
http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2
On Tue, Aug 6, 2013 at 4:49 AM, Richard Low <ri...@wentnet.com> wrote:
> On 6 August 2013 08:40, Aaron Morton <aa...@thelastpickle.com> wrote:
>
>> The reason for me looking at virtual nodes is because of terrible
>> experiences we had with 0.8 repairs and as per documentation (an logically)
>> the virtual nodes seems like it will help repairs being smoother. Is this
>> true?
>>
>> I've not thought too much about how they help repair run smoother, what
>> was the documentation you read ?
>>
>
> There might be a slight improvement but I haven't observed any. The
> difference might be that, because every node shares replicas with every
> other (with high probability), a single repair operation does the same work
> on the node it was called on, but the rest is spread out over the cluster,
> rather than just the RF nodes either side of the repairing node. This
> means the post-repair compaction work will take less time and the length of
> time a node is loaded for during repair is less.
>
> However, the other benefits of vnodes are likely to be much more useful.
>
> Richard.
>
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Posted by Richard Low <ri...@wentnet.com>.
On 6 August 2013 08:40, Aaron Morton <aa...@thelastpickle.com> wrote:
> The reason for me looking at virtual nodes is because of terrible
> experiences we had with 0.8 repairs and as per documentation (an logically)
> the virtual nodes seems like it will help repairs being smoother. Is this
> true?
>
> I've not thought too much about how they help repair run smoother, what
> was the documentation you read ?
>
There might be a slight improvement but I haven't observed any. The
difference might be that, because every node shares replicas with every
other (with high probability), a single repair operation does the same work
on the node it was called on, but the rest is spread out over the cluster,
rather than just the RF nodes either side of the repairing node. This
means the post-repair compaction work will take less time and the length of
time a node is loaded for during repair is less.
However, the other benefits of vnodes are likely to be much more useful.
Richard.
Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?
Posted by Aaron Morton <aa...@thelastpickle.com>.
> The reason for me looking at virtual nodes is because of terrible experiences we had with 0.8 repairs and as per documentation (an logically) the virtual nodes seems like it will help repairs being smoother. Is this true?
I've not thought too much about how they help repair run smoother, what was the documentation you read ?
> Also how to get the right number of virtual nodes?
Use the default 256
Hope that helps.
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 3/08/2013, at 7:39 AM, rash aroskar <ra...@gmail.com> wrote:
> Thanks for helpful responses. The upgrade from 0.8 to 1.2 is not direct, we have setup test cluster where we did upgrade from 0.8 to 1.1 and then 1.2. Also we will do a whole different cluster with 1.2, the 0.8 cluster will not be upgraded. But the data will be moved from 0.8 cluster to 1.2 cluster.
> The reason for me looking at virtual nodes is because of terrible experiences we had with 0.8 repairs and as per documentation (an logically) the virtual nodes seems like it will help repairs being smoother. Is this true? Also how to get the right number of virtual nodes? David suggested 64 vnodes for 20 machines. Is there a formula or a thought process to be followed to get this number right?
>
>
> On Mon, Jul 29, 2013 at 4:15 AM, aaron morton <aa...@thelastpickle.com> wrote:
> I would *strongly* recommend against upgrading from 0.8 directly to 1.2. Skipping a major version is generally not recommended, skipped 3 would seem like carelessness.
>
>> I second Romain, do the upgrade and make sure the health is good first.
>
> +1 but I would also recommend deciding if you actually need to use virtual nodes. The shuffle process can take a long time and people have had mixed experiences with it.
>
> If you wanted to move to 1.2 and get vNodes I would consider spinning up a new cluster and bulk loading into it. You could do an initial load and then to delta loads using snapshots, there would however be a period of stale data in the new cluster until the last delta snapshot is loaded.
>
> Cheers
>
> -----------------
> Aaron Morton
> Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 27/07/2013, at 3:36 AM, David McNelis <dm...@gmail.com> wrote:
>
>> I second Romain, do the upgrade and make sure the health is good first.
>>
>> If you have or plan to have a large number of nodes, you might consider using fewer than 256 as your initial vnodes amount. I think that number is inflated from reasonable in the docs, as we've had some people talk about potential performance degradation if you have a large number of nodes and a very high number of vnodes, if I had it to do over again, I'd have done 64 vnodes as my default (across 20 nodes).
>>
>> Another thing to be very cognizant of before shuffle is disk space. You *must* have less than 50% used in order to do the shuffle successfully because no data is removed (cleaned) from a node during the shuffle process and the shuffle process essentially doubles the amount of data until you're able to run a clean.
>>
>>
>> On Fri, Jul 26, 2013 at 11:25 AM, Romain HARDOUIN <ro...@urssaf.fr> wrote:
>> Vnodes are a great feature. More nodes are involved during operations such as bootstrap, decommission, etc.
>> DataStax documentation is definitely a must read.
>> That said, If I were you, I'd wait somewhat before to shuffle the ring. I'd focus on cluster upgrade and monitoring the nodes. (number of files handles, memory usage, latency, etc).
>> Upgrading from 0.8 to 1.2 can be tricky, there are so many changes since then. Be careful about compaction strategies you choose and double check the options.
>>
>> Regards,
>> Romain
>>
>> rash aroskar <ra...@gmail.com> a écrit sur 25/07/2013 23:25:11 :
>>
>> > De : rash aroskar <ra...@gmail.com>
>> > A : user@cassandra.apache.org,
>> > Date : 25/07/2013 23:25
>> > Objet : cassandra 1.2.5- virtual nodes (num_token) pros/cons?
>> >
>> > Hi,
>> > I am upgrading my cassandra cluster from 0.8 to 1.2.5.
>> > In cassandra 1.2.5 the 'num_token' attribute confuses me.
>> > I understand that it distributes multiple tokens per node but I am
>> > not clear how that is helpful for performance or load balancing. Can
>> > anyone elaborate? has anyone used this feature and knows its
>> > advantages/disadvantages?
>> >
>> > Thanks,
>> > Rashmi
>>
>
>