You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2010/10/28 16:38:22 UTC

[jira] Created: (CASSANDRA-1674) Repair using abnormally large amounts of disk space, dropping messages

Repair using abnormally large amounts of disk space, dropping messages
----------------------------------------------------------------------

                 Key: CASSANDRA-1674
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
             Project: Cassandra
          Issue Type: Bug
          Components: Core
            Reporter: Jonathan Ellis
             Fix For: 0.7.1


I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.

I have seen repair take more space than expected on 0.6 but nothing this extreme.

Other nodes in the cluster are occasionally logging
WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms

The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space

Posted by "Chip Salzenberg (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932787#action_12932787 ] 

Chip Salzenberg commented on CASSANDRA-1674:
--------------------------------------------

Surely something that can overflow disks, even when below 50% usage, is worthy of fix before 0.7.0 is released...

> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.7.1
>
>         Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1674) Repair using abnormally large amounts of disk space

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-1674:
--------------------------------

    Summary: Repair using abnormally large amounts of disk space  (was: Repair using abnormally large amounts of disk space, dropping messages)

> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.7.1
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1674) Repair using abnormally large amounts of disk space

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-1674:
--------------------------------

    Attachment: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt

After finding the differences for all data held on both nodes, repair was not limiting the repaired area to the ranges that the nodes had in common. jbellis: In your example above, because the nodes had no data in common to start with, the MerkleTree was determining that they were different for (0,0], but rather than intersecting the different range with the range of responsibility they had in common, it was repairing all of (0,0].

Attaching a patch for 0.7 to only repair the intersecting ranges.

> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.7.1
>
>         Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933122#action_12933122 ] 

Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------

committed, but I think there is a bug.  With a similar setup to the above (200K keys instead of 1M), the pre-RF change setup is

{code}
Address         Status State   Load            Token                                       
                                       106239986353888428655683112465158427815    
127.0.0.2       Up     Normal  37.97 MB        21212647344528771789748883276744400257      
127.0.0.3       Up     Normal  18.98 MB        63523312719601176253752035031089272162      
127.0.0.1       Up     Normal  19.05 MB        106239986353888428655683112465158427815     
{code}

post-repair is
{code}
Address         Status State   Load            Token                                       
                                       106239986353888428655683112465158427815    
127.0.0.2       Up     Normal  57.01 MB        21212647344528771789748883276744400257      
127.0.0.3       Up     Normal  56.94 MB        63523312719601176253752035031089272162      
127.0.0.1       Up     Normal  19.07 MB        106239986353888428655683112465158427815     
{code}

So eyeballing it looks reasonable.  But when I kill node 2 and run

{code}$ python contrib/py_stress/stress.py -n 200000 -o read{code}

I get a ton of key-not-found exceptions, indicating that not all the data on 2 got replicated to 3.

> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.6.9, 0.7.0
>
>         Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt, for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space, dropping messages

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925854#action_12925854 ] 

Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------

it doesn't have to be single threaded to drop messages, it just has to not run the verbhandler until RPC_TIMEOUT after the message was received.

In practice though I don't see any place where the verbhandler would be blocking or otherwise slow, including the ones running on MISC.

> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>             Fix For: 0.7.0
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1674) Repair using abnormally large amounts of disk space

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-1674:
--------------------------------

    Attachment: for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
                0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt

Patches for 0.6 and 0.7/trunk: ready for review.

> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.6.9, 0.7.0
>
>         Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt, for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932774#action_12932774 ] 

Stu Hood commented on CASSANDRA-1674:
-------------------------------------

I started rebasing this for 0.6, but noticed some odd behaviour in the 0.6 branch: in what cases would SS.getLocalToken not equal the RHS of SS.getLocalPrimaryRange?

> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.7.1
>
>         Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-1674) Repair using abnormally large amounts of disk space

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-1674:
--------------------------------

    Attachment:     (was: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt)

> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.6.9, 0.7.0
>
>         Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt, for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933147#action_12933147 ] 

Hudson commented on CASSANDRA-1674:
-----------------------------------

Integrated in Cassandra-0.6 #7 (See [https://hudson.apache.org/hudson/job/Cassandra-0.6/7/])
    limit repaired ranges to what the nodes have in common
patch by Stu Hood; reviewed by jbellis for CASSANDRA-1674


> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.6.9, 0.7.0
>
>         Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt, for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (CASSANDRA-1674) Repair using abnormally large amounts of disk space, dropping messages

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis reassigned CASSANDRA-1674:
-----------------------------------------

    Assignee: Stu Hood

Here is one way to reproduce something similar, at least.

I start with 1 node, and put 1M rows on it.  Then I add a 2nd node, then a third, then run cleanup.  So I have 25% 50% 25% of the 1M rows on those 3 machines:

{code}
$ nodetool -h localhost -p 8080 ring
Address         Status State   Load            Token                                       
                                       164074424718159380631425626216484638578    
127.0.0.2       Up     Normal  95.39 MB        36509681143663337709904175976843745575      
127.0.0.1       Up     Normal  190.72 MB       121466565577718822332569289829746132598     
127.0.0.3       Up     Normal  95.4 MB         164074424718159380631425626216484638578     
{code}

I increase the RF to 2, and run repair against .2:

{code}
Address         Status State   Load            Token                                       
                                       164074424718159380631425626216484638578    
127.0.0.2       Up     Normal  381.41 MB       36509681143663337709904175976843745575      
127.0.0.1       Up     Normal  381.41 MB       121466565577718822332569289829746132598     
127.0.0.3       Up     Normal  190.75 MB       164074424718159380631425626216484638578     
{code}

Let's call the ranges of data that .2, .1, and .3 originally had R, G, and B.  Post repair, .2 should have RB but it has RGB.  Node 1 should have GB but it also has RGB.

Cleanup puts things in the expected state:
{code}
Address         Status State   Load            Token                                       
                                       164074424718159380631425626216484638578    
127.0.0.2       Up     Normal  190.74 MB       36509681143663337709904175976843745575      
127.0.0.1       Up     Normal  285.61 MB       121466565577718822332569289829746132598     
127.0.0.3       Up     Normal  190.75 MB       164074424718159380631425626216484638578     
{code}

This is reproducible against 0.6 as well as 0.7.

> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.7.1
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space, dropping messages

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925842#action_12925842 ] 

Stu Hood commented on CASSANDRA-1674:
-------------------------------------

I haven't tested the theory, but various messages are executed in the single threaded MISC stage.

> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>             Fix For: 0.7.0
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932867#action_12932867 ] 

Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------

bq. in what cases would SS.getLocalToken not equal the RHS of SS.getLocalPrimaryRange? 

at a guess, some test is mocking up a TokenMetadata that isn't consistent w/ localtoken

> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.6.9, 0.7.0
>
>         Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933285#action_12933285 ] 

Stu Hood commented on CASSANDRA-1674:
-------------------------------------

This still needs to be committed to trunk.

> Repair using abnormally large amounts of disk space
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Stu Hood
>             Fix For: 0.6.9, 0.7.0
>
>         Attachments: 0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt, for-0.6-0001-Only-repair-the-intersecting-portion-of-a-differing-ra.txt
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space, dropping messages

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925817#action_12925817 ] 

Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------

(this is rc1 snapshot, btw)

> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>             Fix For: 0.7.1
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space, dropping messages

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925838#action_12925838 ] 

Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------

bq. WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms

Created CASSANDRA-1676 for this.

> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>             Fix For: 0.7.0
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-1674) Repair using abnormally large amounts of disk space, dropping messages

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926813#action_12926813 ] 

Jonathan Ellis commented on CASSANDRA-1674:
-------------------------------------------

so we have CASSANDRA-1685 and CASSANDRA-1676 committed now, but I don't think those are the root cause here since for them to cause lots of disk space usage repair would have to retry dropped requests which I don't see it doing.  (is this correct?)

> Repair using abnormally large amounts of disk space, dropping messages
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-1674
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1674
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jonathan Ellis
>             Fix For: 0.7.0
>
>
> I'm watching a repair on a 7 node cluster.  Repair was sent to one node; the node had 18G of data.  No other node has more than 28G.  The node where the repair initiated is now up to 261G with 53/60 AES tasks outstanding.
> I have seen repair take more space than expected on 0.6 but nothing this extreme.
> Other nodes in the cluster are occasionally logging
> WARN [ScheduledTasks:1] 2010-10-28 08:31:14,305 MessagingService.java (line 515) Dropped 7 messages in the last 1000ms
> The cluster is quiesced except for the repair.  Not sure if the dropped messages are contributing the the disk space (b/c of retries?).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.