You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Maxim Potekhin <po...@bnl.gov> on 2011/07/19 13:24:03 UTC
Repair taking a long, long time
We have something of the order of 200GB load on each of 3 machines in a
balanced cluster under 0.8.1.
I started repair about 24hrs ago and did some moderate amount of inserts
since then (a small fraction of
data load). The repair still appears to be running. What could go wrong?
Thanks,
Maxim
Re: How does Cassandra decide when to do a minor compaction?
Posted by Maxim Potekhin <po...@bnl.gov>.
Hello Alexandru,
I just want to have a feel what activity to expect on the cluster.
The load from minor compactions is not overwhelming but it
seems non-negligible.
Maxim
On 1/7/2012 5:12 AM, Alexandru Sicoe wrote:
> Hi Maxim,
> Why do you need to know this?
>
> Cheers,
> Alex
>
> On Sat, Jan 7, 2012 at 10:03 AM, aaron morton <aaron@thelastpickle.com
> <ma...@thelastpickle.com>> wrote:
>
>
> http://www.datastax.com/docs/1.0/operations/tuning#tuning-compaction
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 7/01/2012, at 3:17 PM, Maxim Potekhin wrote:
>
>> The subject says it all -- pointers appreciated.
>>
>> Thanks
>>
>> Maxim
>>
>
>
Re: How does Cassandra decide when to do a minor compaction?
Posted by Alexandru Sicoe <ad...@gmail.com>.
Hi Maxim,
Why do you need to know this?
Cheers,
Alex
On Sat, Jan 7, 2012 at 10:03 AM, aaron morton <aa...@thelastpickle.com>wrote:
>
> http://www.datastax.com/docs/1.0/operations/tuning#tuning-compaction
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 7/01/2012, at 3:17 PM, Maxim Potekhin wrote:
>
> The subject says it all -- pointers appreciated.
>
> Thanks
>
> Maxim
>
>
>
Re: How does Cassandra decide when to do a minor compaction?
Posted by aaron morton <aa...@thelastpickle.com>.
http://www.datastax.com/docs/1.0/operations/tuning#tuning-compaction
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 7/01/2012, at 3:17 PM, Maxim Potekhin wrote:
> The subject says it all -- pointers appreciated.
>
> Thanks
>
> Maxim
>
How does Cassandra decide when to do a minor compaction?
Posted by Maxim Potekhin <po...@bnl.gov>.
The subject says it all -- pointers appreciated.
Thanks
Maxim
Re: Repair taking a long, long time
Posted by aaron morton <aa...@thelastpickle.com>.
The first thing to do is understand what the server is doing.
As Edward said, there are two phases to the repair first the differences are calculated and then they are shared between the neighbours. Lets an a third step, once the neighbour gets the data it has to rebuild the indexes and bloom filter, not huge but lets include it for completeness.
So...
0. Check for ERRORS in the log.
1. check nodetool compactstats , if the Merkle tree build is going on it will say "Validation Compaction". Run it twice and check for progress.
2. check nodetool netstats, this will show which segments of the data are been streamed. Run it twice and check for progress.
3. check nodetool compactstats, if the data has completed streaming and indexes are been built it will say "SSTable build"
Once we know what stage of the repair your server is at it's possible to reason about what is going on.
If you want to dive deeper look for a log messages on the machine you started the repair on from the AnitEntropyService.
Hope that helps.
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 21 Jul 2011, at 02:31, David Boxenhorn wrote:
> As I indicated below (but didn't say specifically) another option is to set read repair chance to 1.0 for all your CFs and loop over all your data, since read triggers a read repair.
>
> On Wed, Jul 20, 2011 at 4:58 PM, Maxim Potekhin <po...@bnl.gov> wrote:
> I can re-load all data that I have in the cluster, from a flat-file cache I have
> on NFS, many times faster than the nodetool repair takes. And that's not
> even accurate because as other noted nodetool repair eats up disk space
> for breakfast and takes more than 24hrs on 200GB data load, at which point
> I have to cancel. That's not acceptable. I simply don't know what to do now.
>
>
>
> On 7/20/2011 8:47 AM, David Boxenhorn wrote:
>>
>> I have this problem too, and I don't understand why.
>>
>> I can repair my nodes very quickly by looping though all my data (when you read your data it does read-repair), but nodetool repair takes forever. I understand that nodetool repair builds merkle trees, etc. etc., so it's a different algorithm, but why can't nodetool repair be smart enough to choose the best algorithm? Also, I don't understand what's special about my data that makes nodetool repair so much slower than looping through all my data.
>>
>>
>> On Wed, Jul 20, 2011 at 12:18 AM, Maxim Potekhin <po...@bnl.gov> wrote:
>> Thanks Edward. I'm told by our IT that the switch connecting the nodes is pretty fast.
>> Seriously, in my house I copy complete DVD images from my bedroom to
>> the living room downstairs via WiFi, and a dozen of GB does not seem like a
>> problem, on dirt cheap hardware (Patriot Box Office).
>>
>> I also have just _one_ column major family but caveat emptor -- 8 indexes attached to
>> it (and there will be more). There is one accounting CF which is small, can't possibly
>> make a difference.
>>
>> By contrast, compaction (as in nodetool) performs quite well on this cluster. I start suspecting some
>> sort of malfunction.
>>
>> Looked at the system log during the "repair", there is some compaction agent doing
>> work that I'm not sure makes sense (and I didn't call for it). Disk utilization all of a sudden goes up to 40%
>> per Ganglia, and stays there, this is pretty silly considering the cluster is IDLE and we have SSDs. No external writes,
>> no reads. There are occasional GC stoppages, but these I can live with.
>>
>> This repair debacle happens 2nd time in a row. Cr@p. I need to go to production soon
>> and that doesn't look good at all. If I can't manage a system that simple (and/or get help
>> on this list) I may have to cut losses i.e. stay with Oracle.
>>
>> Regards,
>>
>> Maxim
>>
>>
>>
>>
>> On 7/19/2011 12:16 PM, Edward Capriolo wrote:
>>
>> Well most SSD's are pretty fast. There is one more to consider. If Cassandra determines nodes are out of sync it has to transfer data across the network. If that is the case you have to look at 'nodetool streams' and determine how much data is being transferred between nodes. There are some open tickets where with larger tables repair is streaming more then it needs to. But even if the transfers are only 10% of your 200GB. Transferring 20 GB is not trivial.
>>
>> If you have multiple keyspaces and column families repair one at a time might make the process more manageable.
>>
>>
>
>
Re: Repair taking a long, long time
Posted by David Boxenhorn <da...@citypath.com>.
As I indicated below (but didn't say specifically) another option is to set
read repair chance to 1.0 for all your CFs and loop over all your data,
since read triggers a read repair.
On Wed, Jul 20, 2011 at 4:58 PM, Maxim Potekhin <po...@bnl.gov> wrote:
> **
> I can re-load all data that I have in the cluster, from a flat-file cache I
> have
> on NFS, many times faster than the nodetool repair takes. And that's not
> even accurate because as other noted nodetool repair eats up disk space
> for breakfast and takes more than 24hrs on 200GB data load, at which point
> I have to cancel. That's not acceptable. I simply don't know what to do
> now.
>
>
>
> On 7/20/2011 8:47 AM, David Boxenhorn wrote:
>
> I have this problem too, and I don't understand why.
>
> I can repair my nodes very quickly by looping though all my data (when you
> read your data it does read-repair), but nodetool repair takes forever. I
> understand that nodetool repair builds merkle trees, etc. etc., so it's a
> different algorithm, but why can't nodetool repair be smart enough to choose
> the best algorithm? Also, I don't understand what's special about my data
> that makes nodetool repair so much slower than looping through all my data.
>
>
> On Wed, Jul 20, 2011 at 12:18 AM, Maxim Potekhin <po...@bnl.gov> wrote:
>
>> Thanks Edward. I'm told by our IT that the switch connecting the nodes is
>> pretty fast.
>> Seriously, in my house I copy complete DVD images from my bedroom to
>> the living room downstairs via WiFi, and a dozen of GB does not seem like
>> a
>> problem, on dirt cheap hardware (Patriot Box Office).
>>
>> I also have just _one_ column major family but caveat emptor -- 8 indexes
>> attached to
>> it (and there will be more). There is one accounting CF which is small,
>> can't possibly
>> make a difference.
>>
>> By contrast, compaction (as in nodetool) performs quite well on this
>> cluster. I start suspecting some
>> sort of malfunction.
>>
>> Looked at the system log during the "repair", there is some compaction
>> agent doing
>> work that I'm not sure makes sense (and I didn't call for it). Disk
>> utilization all of a sudden goes up to 40%
>> per Ganglia, and stays there, this is pretty silly considering the cluster
>> is IDLE and we have SSDs. No external writes,
>> no reads. There are occasional GC stoppages, but these I can live with.
>>
>> This repair debacle happens 2nd time in a row. Cr@p. I need to go to
>> production soon
>> and that doesn't look good at all. If I can't manage a system that simple
>> (and/or get help
>> on this list) I may have to cut losses i.e. stay with Oracle.
>>
>> Regards,
>>
>> Maxim
>>
>>
>>
>>
>> On 7/19/2011 12:16 PM, Edward Capriolo wrote:
>>
>>>
>>> Well most SSD's are pretty fast. There is one more to consider. If
>>> Cassandra determines nodes are out of sync it has to transfer data across
>>> the network. If that is the case you have to look at 'nodetool streams' and
>>> determine how much data is being transferred between nodes. There are some
>>> open tickets where with larger tables repair is streaming more then it needs
>>> to. But even if the transfers are only 10% of your 200GB. Transferring 20 GB
>>> is not trivial.
>>>
>>> If you have multiple keyspaces and column families repair one at a time
>>> might make the process more manageable.
>>>
>>
>>
>
>
Re: Repair taking a long, long time
Posted by Maxim Potekhin <po...@bnl.gov>.
I can re-load all data that I have in the cluster, from a flat-file
cache I have
on NFS, many times faster than the nodetool repair takes. And that's not
even accurate because as other noted nodetool repair eats up disk space
for breakfast and takes more than 24hrs on 200GB data load, at which point
I have to cancel. That's not acceptable. I simply don't know what to do now.
On 7/20/2011 8:47 AM, David Boxenhorn wrote:
> I have this problem too, and I don't understand why.
>
> I can repair my nodes very quickly by looping though all my data (when
> you read your data it does read-repair), but nodetool repair takes
> forever. I understand that nodetool repair builds merkle trees, etc.
> etc., so it's a different algorithm, but why can't nodetool repair be
> smart enough to choose the best algorithm? Also, I don't understand
> what's special about my data that makes nodetool repair so much slower
> than looping through all my data.
>
>
> On Wed, Jul 20, 2011 at 12:18 AM, Maxim Potekhin <potekhin@bnl.gov
> <ma...@bnl.gov>> wrote:
>
> Thanks Edward. I'm told by our IT that the switch connecting the
> nodes is pretty fast.
> Seriously, in my house I copy complete DVD images from my bedroom to
> the living room downstairs via WiFi, and a dozen of GB does not
> seem like a
> problem, on dirt cheap hardware (Patriot Box Office).
>
> I also have just _one_ column major family but caveat emptor -- 8
> indexes attached to
> it (and there will be more). There is one accounting CF which is
> small, can't possibly
> make a difference.
>
> By contrast, compaction (as in nodetool) performs quite well on
> this cluster. I start suspecting some
> sort of malfunction.
>
> Looked at the system log during the "repair", there is some
> compaction agent doing
> work that I'm not sure makes sense (and I didn't call for it).
> Disk utilization all of a sudden goes up to 40%
> per Ganglia, and stays there, this is pretty silly considering the
> cluster is IDLE and we have SSDs. No external writes,
> no reads. There are occasional GC stoppages, but these I can live
> with.
>
> This repair debacle happens 2nd time in a row. Cr@p. I need to go
> to production soon
> and that doesn't look good at all. If I can't manage a system that
> simple (and/or get help
> on this list) I may have to cut losses i.e. stay with Oracle.
>
> Regards,
>
> Maxim
>
>
>
>
> On 7/19/2011 12:16 PM, Edward Capriolo wrote:
>
>
> Well most SSD's are pretty fast. There is one more to
> consider. If Cassandra determines nodes are out of sync it has
> to transfer data across the network. If that is the case you
> have to look at 'nodetool streams' and determine how much data
> is being transferred between nodes. There are some open
> tickets where with larger tables repair is streaming more then
> it needs to. But even if the transfers are only 10% of your
> 200GB. Transferring 20 GB is not trivial.
>
> If you have multiple keyspaces and column families repair one
> at a time might make the process more manageable.
>
>
>
Re: Repair taking a long, long time
Posted by Boris Yen <yu...@gmail.com>.
We also got the same problem when using 0.8.0. As far as I know, there are a
few issues relative to 'repair' has been marked as resolved at 0.8.1. Hope
this could really solve our problem.
On Wed, Jul 20, 2011 at 8:47 PM, David Boxenhorn <da...@citypath.com> wrote:
> I have this problem too, and I don't understand why.
>
> I can repair my nodes very quickly by looping though all my data (when you
> read your data it does read-repair), but nodetool repair takes forever. I
> understand that nodetool repair builds merkle trees, etc. etc., so it's a
> different algorithm, but why can't nodetool repair be smart enough to choose
> the best algorithm? Also, I don't understand what's special about my data
> that makes nodetool repair so much slower than looping through all my data.
>
>
>
> On Wed, Jul 20, 2011 at 12:18 AM, Maxim Potekhin <po...@bnl.gov> wrote:
>
>> Thanks Edward. I'm told by our IT that the switch connecting the nodes is
>> pretty fast.
>> Seriously, in my house I copy complete DVD images from my bedroom to
>> the living room downstairs via WiFi, and a dozen of GB does not seem like
>> a
>> problem, on dirt cheap hardware (Patriot Box Office).
>>
>> I also have just _one_ column major family but caveat emptor -- 8 indexes
>> attached to
>> it (and there will be more). There is one accounting CF which is small,
>> can't possibly
>> make a difference.
>>
>> By contrast, compaction (as in nodetool) performs quite well on this
>> cluster. I start suspecting some
>> sort of malfunction.
>>
>> Looked at the system log during the "repair", there is some compaction
>> agent doing
>> work that I'm not sure makes sense (and I didn't call for it). Disk
>> utilization all of a sudden goes up to 40%
>> per Ganglia, and stays there, this is pretty silly considering the cluster
>> is IDLE and we have SSDs. No external writes,
>> no reads. There are occasional GC stoppages, but these I can live with.
>>
>> This repair debacle happens 2nd time in a row. Cr@p. I need to go to
>> production soon
>> and that doesn't look good at all. If I can't manage a system that simple
>> (and/or get help
>> on this list) I may have to cut losses i.e. stay with Oracle.
>>
>> Regards,
>>
>> Maxim
>>
>>
>>
>>
>> On 7/19/2011 12:16 PM, Edward Capriolo wrote:
>>
>>>
>>> Well most SSD's are pretty fast. There is one more to consider. If
>>> Cassandra determines nodes are out of sync it has to transfer data across
>>> the network. If that is the case you have to look at 'nodetool streams' and
>>> determine how much data is being transferred between nodes. There are some
>>> open tickets where with larger tables repair is streaming more then it needs
>>> to. But even if the transfers are only 10% of your 200GB. Transferring 20 GB
>>> is not trivial.
>>>
>>> If you have multiple keyspaces and column families repair one at a time
>>> might make the process more manageable.
>>>
>>
>>
>
Re: Repair taking a long, long time
Posted by David Boxenhorn <da...@citypath.com>.
I have this problem too, and I don't understand why.
I can repair my nodes very quickly by looping though all my data (when you
read your data it does read-repair), but nodetool repair takes forever. I
understand that nodetool repair builds merkle trees, etc. etc., so it's a
different algorithm, but why can't nodetool repair be smart enough to choose
the best algorithm? Also, I don't understand what's special about my data
that makes nodetool repair so much slower than looping through all my data.
On Wed, Jul 20, 2011 at 12:18 AM, Maxim Potekhin <po...@bnl.gov> wrote:
> Thanks Edward. I'm told by our IT that the switch connecting the nodes is
> pretty fast.
> Seriously, in my house I copy complete DVD images from my bedroom to
> the living room downstairs via WiFi, and a dozen of GB does not seem like a
> problem, on dirt cheap hardware (Patriot Box Office).
>
> I also have just _one_ column major family but caveat emptor -- 8 indexes
> attached to
> it (and there will be more). There is one accounting CF which is small,
> can't possibly
> make a difference.
>
> By contrast, compaction (as in nodetool) performs quite well on this
> cluster. I start suspecting some
> sort of malfunction.
>
> Looked at the system log during the "repair", there is some compaction
> agent doing
> work that I'm not sure makes sense (and I didn't call for it). Disk
> utilization all of a sudden goes up to 40%
> per Ganglia, and stays there, this is pretty silly considering the cluster
> is IDLE and we have SSDs. No external writes,
> no reads. There are occasional GC stoppages, but these I can live with.
>
> This repair debacle happens 2nd time in a row. Cr@p. I need to go to
> production soon
> and that doesn't look good at all. If I can't manage a system that simple
> (and/or get help
> on this list) I may have to cut losses i.e. stay with Oracle.
>
> Regards,
>
> Maxim
>
>
>
>
> On 7/19/2011 12:16 PM, Edward Capriolo wrote:
>
>>
>> Well most SSD's are pretty fast. There is one more to consider. If
>> Cassandra determines nodes are out of sync it has to transfer data across
>> the network. If that is the case you have to look at 'nodetool streams' and
>> determine how much data is being transferred between nodes. There are some
>> open tickets where with larger tables repair is streaming more then it needs
>> to. But even if the transfers are only 10% of your 200GB. Transferring 20 GB
>> is not trivial.
>>
>> If you have multiple keyspaces and column families repair one at a time
>> might make the process more manageable.
>>
>
>
Re: Repair taking a long, long time
Posted by Maxim Potekhin <po...@bnl.gov>.
Thanks Edward. I'm told by our IT that the switch connecting the nodes
is pretty fast.
Seriously, in my house I copy complete DVD images from my bedroom to
the living room downstairs via WiFi, and a dozen of GB does not seem like a
problem, on dirt cheap hardware (Patriot Box Office).
I also have just _one_ column major family but caveat emptor -- 8
indexes attached to
it (and there will be more). There is one accounting CF which is small,
can't possibly
make a difference.
By contrast, compaction (as in nodetool) performs quite well on this
cluster. I start suspecting some
sort of malfunction.
Looked at the system log during the "repair", there is some compaction
agent doing
work that I'm not sure makes sense (and I didn't call for it). Disk
utilization all of a sudden goes up to 40%
per Ganglia, and stays there, this is pretty silly considering the
cluster is IDLE and we have SSDs. No external writes,
no reads. There are occasional GC stoppages, but these I can live with.
This repair debacle happens 2nd time in a row. Cr@p. I need to go to
production soon
and that doesn't look good at all. If I can't manage a system that
simple (and/or get help
on this list) I may have to cut losses i.e. stay with Oracle.
Regards,
Maxim
On 7/19/2011 12:16 PM, Edward Capriolo wrote:
>
> Well most SSD's are pretty fast. There is one more to consider. If
> Cassandra determines nodes are out of sync it has to transfer data
> across the network. If that is the case you have to look at 'nodetool
> streams' and determine how much data is being transferred between
> nodes. There are some open tickets where with larger tables repair is
> streaming more then it needs to. But even if the transfers are only
> 10% of your 200GB. Transferring 20 GB is not trivial.
>
> If you have multiple keyspaces and column families repair one at a
> time might make the process more manageable.
Re: Repair taking a long, long time
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Jul 19, 2011 at 11:36 AM, Maxim Potekhin <po...@bnl.gov> wrote:
> Thanks for pointer. As a comment to your disk speed comment --
> I had already installed SSDs on these nodes.
>
> Maxim
>
>
> On 7/19/2011 9:41 AM, Edward Capriolo wrote:
>
>
>
> On Tue, Jul 19, 2011 at 7:24 AM, Maxim Potekhin <po...@bnl.gov> wrote:
>
>> We have something of the order of 200GB load on each of 3 machines in a
>> balanced cluster under 0.8.1.
>> I started repair about 24hrs ago and did some moderate amount of inserts
>> since then (a small fraction of
>> data load). The repair still appears to be running. What could go wrong?
>>
>> Thanks,
>> Maxim
>>
>>
>
> Repair calculates a merkle tree on each node and then transmits differences
> to its neighbours. Because this process run on all your data it can take a
> LONG time.
> See http://wiki.apache.org/cassandra/LargeDataSetConsiderations
>
> Use 'nodetool compactionstats' and 'nodetool streams' to check the
> progress. My rule of thumb is do not have more storage on a node then you
> can compact in 3 hours. So for that you either need fast disks or more
> nodes.
>
>
>
Well most SSD's are pretty fast. There is one more to consider. If Cassandra
determines nodes are out of sync it has to transfer data across the network.
If that is the case you have to look at 'nodetool streams' and determine how
much data is being transferred between nodes. There are some open tickets
where with larger tables repair is streaming more then it needs to. But even
if the transfers are only 10% of your 200GB. Transferring 20 GB is not
trivial.
If you have multiple keyspaces and column families repair one at a time
might make the process more manageable.
Re: Repair taking a long, long time
Posted by Maxim Potekhin <po...@bnl.gov>.
Thanks for pointer. As a comment to your disk speed comment --
I had already installed SSDs on these nodes.
Maxim
On 7/19/2011 9:41 AM, Edward Capriolo wrote:
>
>
> On Tue, Jul 19, 2011 at 7:24 AM, Maxim Potekhin <potekhin@bnl.gov
> <ma...@bnl.gov>> wrote:
>
> We have something of the order of 200GB load on each of 3 machines
> in a balanced cluster under 0.8.1.
> I started repair about 24hrs ago and did some moderate amount of
> inserts since then (a small fraction of
> data load). The repair still appears to be running. What could go
> wrong?
>
> Thanks,
> Maxim
>
>
>
> Repair calculates a merkle tree on each node and then transmits
> differences to its neighbours. Because this process run on all your
> data it can take a LONG time.
> See http://wiki.apache.org/cassandra/LargeDataSetConsiderations
>
> Use 'nodetool compactionstats' and 'nodetool streams' to check the
> progress. My rule of thumb is do not have more storage on a node then
> you can compact in 3 hours. So for that you either need fast disks or
> more nodes.
>
Re: Repair taking a long, long time
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Jul 19, 2011 at 7:24 AM, Maxim Potekhin <po...@bnl.gov> wrote:
> We have something of the order of 200GB load on each of 3 machines in a
> balanced cluster under 0.8.1.
> I started repair about 24hrs ago and did some moderate amount of inserts
> since then (a small fraction of
> data load). The repair still appears to be running. What could go wrong?
>
> Thanks,
> Maxim
>
>
Repair calculates a merkle tree on each node and then transmits differences
to its neighbours. Because this process run on all your data it can take a
LONG time.
See http://wiki.apache.org/cassandra/LargeDataSetConsiderations
Use 'nodetool compactionstats' and 'nodetool streams' to check the progress.
My rule of thumb is do not have more storage on a node then you can compact
in 3 hours. So for that you either need fast disks or more nodes.