You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2010/10/28 16:38:22 UTC
[jira] Created: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space, dropping messages
Repair using abnormally large amounts of disk space, dropping messages
----------------------------------------------------------------------
Key: CASSANDRA-1674
URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Jonathan Ellis
Fix For: 0.7.1
I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
I have seen repair take more space than expected on 0.6 but nothing this extreme.
Other nodes in the cluster are occasionally logging
WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space
Posted by "Chip Salzenberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932787#action_12932787 ]
Chip Salzenberg commented on CASSANDRA-1674:
--------------------------------------------
Surely something that can overflow disks, even when below 50% usage, is worthy of fix before 0.7.0 is released...
> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.7.1
>
> Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-1674:
--------------------------------
Summary: Repair using abnormally large amounts of disk space (was: Repair using abnormally large amounts of disk space, dropping messages)
> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.7.1
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-1674:
--------------------------------
Attachment: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
After finding the differences for all data held on both nodes, repair was not limiting the repaired area to the ranges that the nodes had in common. jbellis: In your example above, because the nodes had no data in common to start with, the MerkleTree was determining that they were different for (0,0], but rather than intersecting the different range with the range of responsibility they had in common, it was repairing all of (0,0].
Attaching a patch for 0.7 to only repair the intersecting ranges.
> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.7.1
>
> Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933122#action_12933122 ]
Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------
committed, but I think there is a bug. With a similar setup to the above (200K keys instead of 1M), the pre-RF change setup is
{code}
Address Status State Load Token
106239986353888428655683112465158427815
127.0.0.2 Up Normal 37.97 MB 21212647344528771789748883276744400257
127.0.0.3 Up Normal 18.98 MB 63523312719601176253752035031089272162
127.0.0.1 Up Normal 19.05 MB 106239986353888428655683112465158427815
{code}
post-repair is
{code}
Address Status State Load Token
106239986353888428655683112465158427815
127.0.0.2 Up Normal 57.01 MB 21212647344528771789748883276744400257
127.0.0.3 Up Normal 56.94 MB 63523312719601176253752035031089272162
127.0.0.1 Up Normal 19.07 MB 106239986353888428655683112465158427815
{code}
So eyeballing it looks reasonable. But when I kill node 2 and run
{code}$ python contrib/py_stress/stress.py -n 200000 -o read{code}
I get a ton of key-not-found exceptions, indicating that not all the data on 2 got replicated to 3.
> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.6.9, 0.7.0
>
> Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt, for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space, dropping messages
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925854#action_12925854 ]
Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------
it doesn't have to be single threaded to drop messages, it just has to not run the verbhandler until RPC_TIMEOUT after the message was received.
In practice though I don't see any place where the verbhandler would be blocking or otherwise slow, including the ones running on MISC.
> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7.0
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-1674:
--------------------------------
Attachment: for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
Patches for 0.6 and 0.7/trunk: ready for review.
> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.6.9, 0.7.0
>
> Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt, for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932774#action_12932774 ]
Stu Hood commented on CASSANDRA-1674:
-------------------------------------
I started rebasing this for 0.6, but noticed some odd behaviour in the 0.6 branch: in what cases would SS.getLocalToken not equal the RHS of SS.getLocalPrimaryRange?
> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.7.1
>
> Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-1674:
--------------------------------
Attachment: (was: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt)
> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.6.9, 0.7.0
>
> Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt, for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933147#action_12933147 ]
Hudson commented on CASSANDRA-1674:
-----------------------------------
Integrated in Cassandra-0.6 #7 (See [https://hudson.apache.org/hudson/job/Cassandra-0.6/7/])
limit repaired ranges to what the nodes have in common
patch by Stu Hood; reviewed by jbellis for CASSANDRA-1674
> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.6.9, 0.7.0
>
> Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt, for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space, dropping messages
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-1674:
-----------------------------------------
Assignee: Stu Hood
Here is one way to reproduce something similar, at least.
I start with 1 node, and put 1M rows on it. Then I add a 2nd node, then a third, then run cleanup. So I have 25% 50% 25% of the 1M rows on those 3 machines:
{code}
$ nodetool -h localhost -p 8080 ring
Address Status State Load Token
164074424718159380631425626216484638578
127.0.0.2 Up Normal 95.39 MB 36509681143663337709904175976843745575
127.0.0.1 Up Normal 190.72 MB 121466565577718822332569289829746132598
127.0.0.3 Up Normal 95.4 MB 164074424718159380631425626216484638578
{code}
I increase the RF to 2, and run repair against .2:
{code}
Address Status State Load Token
164074424718159380631425626216484638578
127.0.0.2 Up Normal 381.41 MB 36509681143663337709904175976843745575
127.0.0.1 Up Normal 381.41 MB 121466565577718822332569289829746132598
127.0.0.3 Up Normal 190.75 MB 164074424718159380631425626216484638578
{code}
Let's call the ranges of data that .2, .1, and .3 originally had R, G, and B. Post repair, .2 should have RB but it has RGB. Node 1 should have GB but it also has RGB.
Cleanup puts things in the expected state:
{code}
Address Status State Load Token
164074424718159380631425626216484638578
127.0.0.2 Up Normal 190.74 MB 36509681143663337709904175976843745575
127.0.0.1 Up Normal 285.61 MB 121466565577718822332569289829746132598
127.0.0.3 Up Normal 190.75 MB 164074424718159380631425626216484638578
{code}
This is reproducible against 0.6 as well as 0.7.
> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.7.1
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space, dropping messages
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925842#action_12925842 ]
Stu Hood commented on CASSANDRA-1674:
-------------------------------------
I haven't tested the theory, but various messages are executed in the single threaded MISC stage.
> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7.0
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932867#action_12932867 ]
Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------
bq. in what cases would SS.getLocalToken not equal the RHS of SS.getLocalPrimaryRange?
at a guess, some test is mocking up a TokenMetadata that isn't consistent w/ localtoken
> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.6.9, 0.7.0
>
> Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933285#action_12933285 ]
Stu Hood commented on CASSANDRA-1674:
-------------------------------------
This still needs to be committed to trunk.
> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Stu Hood
> Fix For: 0.6.9, 0.7.0
>
> Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt, for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space, dropping messages
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925817#action_12925817 ]
Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------
(this is rc1 snapshot, btw)
> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7.1
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space, dropping messages
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925838#action_12925838 ]
Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------
bq. WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
Created CASSANDRA-1676 for this.
> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7.0
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1674) Repair using abnormally large
amounts of disk space, dropping messages
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926813#action_12926813 ]
Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------
so we have CASSANDRA-1685 and CASSANDRA-1676 committed now, but I don't think those are the root cause here since for them to cause lots of disk space usage repair would have to retry dropped requests which I don't see it doing. (is this correct?)
> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-1674
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Fix For: 0.7.0
>
>
> I'm watching a repair on a 7 node cluster. Repair was sent to one node; the node had 18G of data. No other node has more than 28G. The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair. Not sure if the dropped messages are contributing the the disk space (b/c of retries?).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.