You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by John Watson <jo...@disqus.com> on 2013/04/28 23:21:20 UTC
cassandra-shuffle time to completion and required disk space
The amount of time/space cassandra-shuffle requires when upgrading to using
vnodes should really be apparent in documentation (when some is made).
Only semi-noticeable remark about the exorbitant amount of time is a bullet
point in: http://wiki.apache.org/cassandra/VirtualNodes/Balance
"Shuffling will entail moving a lot of data around the cluster and so has
the potential to consume a lot of disk and network I/O, and to take a
considerable amount of time. For this to be an online operation, the
shuffle will need to operate on a lower priority basis to other streaming
operations, and should be expected to take days or weeks to complete."
We tried running shuffle on a QA version of our cluster and 2 things were
brought to light:
- Even with no reads/writes it was going to take 20 days
- Each machine needed enough free diskspace to potentially hold the entire
cluster's sstables on disk
Regards,
John
Re: cassandra-shuffle time to completion and required disk space
Posted by John Watson <jo...@disqus.com>.
11 nodes
1 keyspace
256 vnodes per node
upgraded 1.1.9 to 1.2.3 a week ago
These are taken just before starting shuffle (ran repair/cleanup the day
before).
During shuffle disabled all reads/writes to the cluster.
nodetool status keyspace:
Load Tokens Owns (effective) Host ID
80.95 GB 256 16.7% 754f9f4c-4ba7-4495-97e7-1f5b6755cb27
87.15 GB 256 16.7% 93f4400a-09d9-4ca0-b6a6-9bcca2427450
98.16 GB 256 16.7% ff821e8e-b2ca-48a9-ac3f-8234b16329ce
142.6 GB 253 100.0% 339c474f-cf19-4ada-9a47-8b10912d5eb3
77.64 GB 256 16.7% e59a02b3-8b91-4abd-990e-b3cb2a494950
194.31 GB 256 25.0% 6d726cbf-147d-426e-a735-e14928c95e45
221.94 GB 256 33.3% 83ca527c-60c5-4ea0-89a8-de53b92b99c8
87.61 GB 256 16.7% c3ea4026-551b-4a14-a346-480e8c1fe283
101.02 GB 256 16.7% df7ba879-74ad-400b-b371-91b45dcbed37
172.44 GB 256 25.0% 78192d73-be0b-4d49-a129-9bec0770efed
108.5 GB 256 16.7% 9889280a-1433-439e-bb84-6b7e7f44d761
nodetool status:
Load Tokens Owns Host ID
142.6 GB 253 97.5% 339c474f-cf19-4ada-9a47-8b10912d5eb3
172.44 GB 256 0.1% 78192d73-be0b-4d49-a129-9bec0770efed
221.94 GB 256 0.4% 83ca527c-60c5-4ea0-89a8-de53b92b99c8
194.31 GB 256 0.1% 6d726cbf-147d-426e-a735-e14928c95e45
77.64 GB 256 0.3% e59a02b3-8b91-4abd-990e-b3cb2a494950
87.15 GB 256 0.4% 93f4400a-09d9-4ca0-b6a6-9bcca2427450
98.16 GB 256 0.1% ff821e8e-b2ca-48a9-ac3f-8234b16329ce
87.61 GB 256 0.3% c3ea4026-551b-4a14-a346-480e8c1fe283
80.95 GB 256 0.4% 754f9f4c-4ba7-4495-97e7-1f5b6755cb27
108.5 GB 256 0.1% 9889280a-1433-439e-bb84-6b7e7f44d761
101.02 GB 256 0.3% df7ba879-74ad-400b-b371-91b45dcbed37
Here's image of the actual disk usage during shuffle:
https://dl.dropbox.com/s/bx57j1z5c2spqo0/shuffle%20disk%20space.png
Little after 00:00 I disabled/cleared the xfers and restarted the cluster
(those drops around 00:15 are the restarts) before starting running
cleanup. The disks are only 540G and whenever cassandra runs out of disk
space, bad things seem to happen. Was just barely able to run cleanup
without running out space after the failed shuffle.
After the restart:
Load Tokens Owns (effective) Host ID
131.73 GB 256 16.7% 754f9f4c-4ba7-4495-97e7-1f5b6755cb27
418.88 GB 255 16.7% 93f4400a-09d9-4ca0-b6a6-9bcca2427450
171.19 GB 255 8.5% ff821e8e-b2ca-48a9-ac3f-8234b16329ce
142.61 GB 253 100.0% 339c474f-cf19-4ada-9a47-8b10912d5eb3
178.83 GB 257 24.9% e59a02b3-8b91-4abd-990e-b3cb2a494950
442.32 GB 257 25.0% 6d726cbf-147d-426e-a735-e14928c95e45
185.28 GB 257 16.7% c3ea4026-551b-4a14-a346-480e8c1fe283
274.47 GB 255 33.3% 83ca527c-60c5-4ea0-89a8-de53b92b99c8
210.73 GB 256 16.7% df7ba879-74ad-400b-b371-91b45dcbed37
274.49 GB 256 25.0% 78192d73-be0b-4d49-a129-9bec0770efed
106.47 GB 256 16.7% 9889280a-1433-439e-bb84-6b7e7f44d761
It's currently still running cleanup, so taking the output from status will
be a little inaccurate.
I have everything instrumented by Metrics being pushed into Graphite. So if
there's graphs/data that may help from there please let me know.
Thanks,
John
On Sun, Apr 28, 2013 at 2:52 PM, aaron morton <aa...@thelastpickle.com>wrote:
> Can you provide some info on the number of nodes, node load, cluster load
> etc ?
>
> AFAIK shuffle was not an easy thing to test and does not get much real
> world use as only some people will run it and they (normally) use it once.
>
> Any info you can provide may help improve the process.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 29/04/2013, at 9:21 AM, John Watson <jo...@disqus.com> wrote:
>
> The amount of time/space cassandra-shuffle requires when upgrading to
> using vnodes should really be apparent in documentation (when some is made).
>
> Only semi-noticeable remark about the exorbitant amount of time is a
> bullet point in: http://wiki.apache.org/cassandra/VirtualNodes/Balance
>
> "Shuffling will entail moving a lot of data around the cluster and so has
> the potential to consume a lot of disk and network I/O, and to take a
> considerable amount of time. For this to be an online operation, the
> shuffle will need to operate on a lower priority basis to other streaming
> operations, and should be expected to take days or weeks to complete."
>
> We tried running shuffle on a QA version of our cluster and 2 things were
> brought to light:
> - Even with no reads/writes it was going to take 20 days
> - Each machine needed enough free diskspace to potentially hold the
> entire cluster's sstables on disk
>
> Regards,
>
> John
>
>
>
Re: cassandra-shuffle time to completion and required disk space
Posted by aaron morton <aa...@thelastpickle.com>.
Can you provide some info on the number of nodes, node load, cluster load etc ?
AFAIK shuffle was not an easy thing to test and does not get much real world use as only some people will run it and they (normally) use it once.
Any info you can provide may help improve the process.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 29/04/2013, at 9:21 AM, John Watson <jo...@disqus.com> wrote:
> The amount of time/space cassandra-shuffle requires when upgrading to using vnodes should really be apparent in documentation (when some is made).
>
> Only semi-noticeable remark about the exorbitant amount of time is a bullet point in: http://wiki.apache.org/cassandra/VirtualNodes/Balance
>
> "Shuffling will entail moving a lot of data around the cluster and so has the potential to consume a lot of disk and network I/O, and to take a considerable amount of time. For this to be an online operation, the shuffle will need to operate on a lower priority basis to other streaming operations, and should be expected to take days or weeks to complete."
>
> We tried running shuffle on a QA version of our cluster and 2 things were brought to light:
> - Even with no reads/writes it was going to take 20 days
> - Each machine needed enough free diskspace to potentially hold the entire cluster's sstables on disk
>
> Regards,
>
> John
Re: cassandra-shuffle time to completion and required disk space
Posted by Richard Low <ri...@wentnet.com>.
Hi John,
> - Each machine needed enough free diskspace to potentially hold the
entire cluster's sstables on disk
I wrote a possible explanation for why Cassandra is trying to use too much
space on your ticket:
https://issues.apache.org/jira/browse/CASSANDRA-5525
if you could provide the information there we can hopefully fix it.
Richard.
Re: cassandra-shuffle time to completion and required disk space
Posted by aaron morton <aa...@thelastpickle.com>.
> These are taken just before starting shuffle (ran repair/cleanup the day before).
> During shuffle disabled all reads/writes to the cluster.
>
> nodetool status keyspace:
>
> Load Tokens Owns (effective) Host ID
> 80.95 GB 256 16.7% 754f9f4c-4ba7-4495-97e7-1f5b6755cb27
I'm a little confused when nodetool status was showing 256 tokens before the shuffle was run.
Did you set num_tokens during the upgrade process ? But I doubt that would change anything as the inital_token is set in the system tables during bootstrap.
`bin/cassandra-shuffle ls` will show the list of moves the shuffle process was/is going to run. What does that say?
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 30/04/2013, at 5:08 AM, John Watson <jo...@disqus.com> wrote:
> That's what we tried first before the shuffle. And ran into the space issue.
>
> That's detailed in another thread title: "Adding nodes in 1.2 with vnodes requires huge disks"
>
>
> On Mon, Apr 29, 2013 at 4:08 AM, Sam Overton <sa...@acunu.com> wrote:
> An alternative to running shuffle is to do a rolling bootstrap/decommission. You would set num_tokens on the existing hosts (and restart them) so that they split their ranges, then bootstrap in N new hosts, then decommission the old ones.
>
>
>
> On 28 April 2013 22:21, John Watson <jo...@disqus.com> wrote:
> The amount of time/space cassandra-shuffle requires when upgrading to using vnodes should really be apparent in documentation (when some is made).
>
> Only semi-noticeable remark about the exorbitant amount of time is a bullet point in: http://wiki.apache.org/cassandra/VirtualNodes/Balance
>
> "Shuffling will entail moving a lot of data around the cluster and so has the potential to consume a lot of disk and network I/O, and to take a considerable amount of time. For this to be an online operation, the shuffle will need to operate on a lower priority basis to other streaming operations, and should be expected to take days or weeks to complete."
>
> We tried running shuffle on a QA version of our cluster and 2 things were brought to light:
> - Even with no reads/writes it was going to take 20 days
> - Each machine needed enough free diskspace to potentially hold the entire cluster's sstables on disk
>
> Regards,
>
> John
>
>
>
> --
> Sam Overton
> Acunu | http://www.acunu.com | @acunu
>
Re: cassandra-shuffle time to completion and required disk space
Posted by John Watson <jo...@disqus.com>.
That's what we tried first before the shuffle. And ran into the space issue.
That's detailed in another thread title: "Adding nodes in 1.2 with vnodes
requires huge disks"
On Mon, Apr 29, 2013 at 4:08 AM, Sam Overton <sa...@acunu.com> wrote:
> An alternative to running shuffle is to do a rolling
> bootstrap/decommission. You would set num_tokens on the existing hosts (and
> restart them) so that they split their ranges, then bootstrap in N new
> hosts, then decommission the old ones.
>
>
>
> On 28 April 2013 22:21, John Watson <jo...@disqus.com> wrote:
>
>> The amount of time/space cassandra-shuffle requires when upgrading to
>> using vnodes should really be apparent in documentation (when some is made).
>>
>> Only semi-noticeable remark about the exorbitant amount of time is a
>> bullet point in: http://wiki.apache.org/cassandra/VirtualNodes/Balance
>>
>> "Shuffling will entail moving a lot of data around the cluster and so has
>> the potential to consume a lot of disk and network I/O, and to take a
>> considerable amount of time. For this to be an online operation, the
>> shuffle will need to operate on a lower priority basis to other streaming
>> operations, and should be expected to take days or weeks to complete."
>>
>> We tried running shuffle on a QA version of our cluster and 2 things were
>> brought to light:
>> - Even with no reads/writes it was going to take 20 days
>> - Each machine needed enough free diskspace to potentially hold the
>> entire cluster's sstables on disk
>>
>> Regards,
>>
>> John
>>
>
>
>
> --
> Sam Overton
> Acunu | http://www.acunu.com | @acunu
>
Re: cassandra-shuffle time to completion and required disk space
Posted by Sam Overton <sa...@acunu.com>.
An alternative to running shuffle is to do a rolling
bootstrap/decommission. You would set num_tokens on the existing hosts (and
restart them) so that they split their ranges, then bootstrap in N new
hosts, then decommission the old ones.
On 28 April 2013 22:21, John Watson <jo...@disqus.com> wrote:
> The amount of time/space cassandra-shuffle requires when upgrading to
> using vnodes should really be apparent in documentation (when some is made).
>
> Only semi-noticeable remark about the exorbitant amount of time is a
> bullet point in: http://wiki.apache.org/cassandra/VirtualNodes/Balance
>
> "Shuffling will entail moving a lot of data around the cluster and so has
> the potential to consume a lot of disk and network I/O, and to take a
> considerable amount of time. For this to be an online operation, the
> shuffle will need to operate on a lower priority basis to other streaming
> operations, and should be expected to take days or weeks to complete."
>
> We tried running shuffle on a QA version of our cluster and 2 things were
> brought to light:
> - Even with no reads/writes it was going to take 20 days
> - Each machine needed enough free diskspace to potentially hold the
> entire cluster's sstables on disk
>
> Regards,
>
> John
>
--
Sam Overton
Acunu | http://www.acunu.com | @acunu