You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jianing hu (JIRA)" <ji...@apache.org> on 2010/03/29 08:01:29 UTC

[jira] Created: (CASSANDRA-924) lost data replica not restored after server restart

lost data replica not restored after server restart
---------------------------------------------------

                 Key: CASSANDRA-924
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
             Project: Cassandra
          Issue Type: Bug
    Affects Versions: 0.5
         Environment: CentOS 5.2
            Reporter: Jianing hu


With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
Steps to reproduce:
1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
on cs3. So far so good
3) Bring down cs3 and wipe out its data directory
4) Bring up cs3
5) run nodeprobe repair Keyspace1 on cs3, the flush
At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Jianing hu (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jianing hu updated CASSANDRA-924:
---------------------------------

    Attachment: cs3.log
                cs2.log
                cs1.log

still no luck. log files attached.

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924-0.5.patch, 924.patch, 924.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) incorrect neighbor calculation in repair

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-924:
-------------------------------------

    Summary: incorrect neighbor calculation in repair  (was: lost data replica not restored after server restart)

> incorrect neighbor calculation in repair
> ----------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>            Assignee: Stu Hood
>             Fix For: 0.6.1
>
>         Attachments: 0001-Calculate-neighbors-using-getRangeToEndpointMap.patch, 0002-Always-trigger-streaming-repairs.patch, 0003-Unit-test-for-AEService-neighbor-calculation.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-924:
-------------------------------------

          Component/s: Core
    Affects Version/s:     (was: 0.6.1)
                           (was: 0.7)
                           (was: 0.6)
                           (was: 0.5)
        Fix Version/s:     (was: 0.7)
                           (was: 0.6)
                           (was: 0.5)
             Assignee: Stu Hood

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>            Assignee: Stu Hood
>             Fix For: 0.6.1
>
>         Attachments: 924-0.5.patch, 924.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-924:
-------------------------------

    Affects Version/s: 0.7
                       0.6.1
                       0.6
        Fix Version/s: 0.7
                       0.6.1
                       0.6
                       0.5

This one is pretty serious... apparently AntiEntropyService is always calculating who to initiate repairs with using getNaturalEndpoints(mytoken), so when RF is less than the number of nodes, it is likely that trees are only sent in one direction (clockwise around the ring). Additionally, we need to do the inverse of getNaturalEndpoints, and determine which other nodes we are holding replicas for.

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Jianing hu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851212#action_12851212 ] 

Jianing hu commented on CASSANDRA-924:
--------------------------------------

Sorry my bad. I was indeed using the wrong patch in my last test (wget renamed the file). The latest patch works!

Thank you so much for bearing with me and getting it fixed so fast.

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-924:
-------------------------------

    Attachment: 924.patch

Uses getRangeToAddressMap to calculate all endpoints that node A is storing replicas for, in addition to endpoints storing replicas for node A.

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924.patch, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-924:
-------------------------------

    Attachment: 924-0.5.patch
                924.patch

Patches updated to remove stubbed out 'range read-repair' option.

Jianing: can you give your test one more try?

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924-0.5.patch, 924.patch, 924.patch, cs1.log, cs2.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-924) incorrect neighbor calculation in repair

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853510#action_12853510 ] 

Stu Hood commented on CASSANDRA-924:
------------------------------------

> why does the test patch add endpoints.add to forceTableRepair?
Rather than implementing the neighbor calculation in two places, I wanted to reuse the logic in AES. In addition to the neighbors, forceTableRepair sends a tree request to localhost, so we add it back to the neighbor list.

> incorrect neighbor calculation in repair
> ----------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>            Assignee: Stu Hood
>             Fix For: 0.6.1
>
>         Attachments: 0001-Calculate-neighbors-using-getRangeToEndpointMap.patch, 0002-Always-trigger-streaming-repairs.patch, 0003-Unit-test-for-AEService-neighbor-calculation.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Jianing hu (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jianing hu updated CASSANDRA-924:
---------------------------------

    Attachment: cs3.log
                cs2.log
                cs1.log

still doesn't work for me. i've attached logs from all servers.
(i added the "get neighbors" log just to be sure i was running the patched code)

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924.patch, cs1.log, cs2.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851214#action_12851214 ] 

Jonathan Ellis commented on CASSANDRA-924:
------------------------------------------

is this something we can add a unit test for?

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Jianing hu (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jianing hu updated CASSANDRA-924:
---------------------------------

    Attachment: error.log
                storage-conf.xml

conf file used in the test and error log

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>         Attachments: error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Jianing hu (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jianing hu updated CASSANDRA-924:
---------------------------------

    Attachment: error.log

log file from cs2

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>         Attachments: error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Roger Schildmeijer (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850991#action_12850991 ] 

Roger Schildmeijer commented on CASSANDRA-924:
----------------------------------------------

I could be wrong but isn't this exception very simliar to the "upgrade to jdk build 18"-problem?

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>         Attachments: error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Jianing hu (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jianing hu updated CASSANDRA-924:
---------------------------------

    Attachment: error.log

log file from cs1

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>         Attachments: error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-924:
-------------------------------

    Attachment:     (was: 924.patch)

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>            Assignee: Stu Hood
>             Fix For: 0.6.1
>
>         Attachments: cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-924:
-------------------------------

    Attachment:     (was: 924-0.5.patch)

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851205#action_12851205 ] 

Stu Hood commented on CASSANDRA-924:
------------------------------------

I think you might have used the wrong version of the patch (I see log entries that I removed in the most recent version). I just deleted the older version, and I've tested that this version works on a 3 node cluster with RF=2.

Very sorry to use you like a guinea pig like that, but thank you very much for trying out each version.

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851165#action_12851165 ] 

Stu Hood commented on CASSANDRA-924:
------------------------------------

Yea, it's a naive comparison based only on the trees. But fixing the stubbed out 'range read-repair' option is much easier.

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924.patch, cs1.log, cs2.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851145#action_12851145 ] 

Stu Hood commented on CASSANDRA-924:
------------------------------------

Soo, we resolved one issue, and exposed another. Because less than 5% of possible keys (in the UTF-8 space) were out of sync between the nodes, AntiEntropyService decided not to do a repair.

I'll update the patches with a solution this evening. Thanks for your patience!

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924.patch, cs1.log, cs2.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851161#action_12851161 ] 

Jonathan Ellis commented on CASSANDRA-924:
------------------------------------------

is the problem that it's guessing for % of "utf8 space" instead of "keys actually on the servers?"

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924.patch, cs1.log, cs2.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-924:
-------------------------------

    Attachment: 0003-Unit-test-for-AEService-neighbor-calculation.patch
                0002-Always-trigger-streaming-repairs.patch
                0001-Calculate-neighbors-using-getRangeToEndpointMap.patch

Patchset for 0.6 and trunk, including a test for neighbor calculation.

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>            Assignee: Stu Hood
>             Fix For: 0.6.1
>
>         Attachments: 0001-Calculate-neighbors-using-getRangeToEndpointMap.patch, 0002-Always-trigger-streaming-repairs.patch, 0003-Unit-test-for-AEService-neighbor-calculation.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850926#action_12850926 ] 

Stu Hood commented on CASSANDRA-924:
------------------------------------

Do you have the logs from one of the other machines? AntiEntropyService on the failed box appears to have given the correct result, but both of the other boxes should have had tree mismatches.

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>         Attachments: error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-924:
-------------------------------

    Attachment:     (was: 924-0.5.patch)

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>            Assignee: Stu Hood
>             Fix For: 0.6.1
>
>         Attachments: cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (CASSANDRA-924) incorrect neighbor calculation in repair

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853505#action_12853505 ] 

Jonathan Ellis commented on CASSANDRA-924:
------------------------------------------

why does the test patch add

endpoints.add(FBUtilities.getLocalAddress());

to forceTableRepair?

> incorrect neighbor calculation in repair
> ----------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>            Assignee: Stu Hood
>             Fix For: 0.6.1
>
>         Attachments: 0001-Calculate-neighbors-using-getRangeToEndpointMap.patch, 0002-Always-trigger-streaming-repairs.patch, 0003-Unit-test-for-AEService-neighbor-calculation.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-924:
-------------------------------

    Attachment: 924-0.5.patch

Backport of the 924.patch for 0.5.

Jianing: would you mind testing the appropriate patch for your version?

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924.patch, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (CASSANDRA-924) lost data replica not restored after server restart

Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stu Hood updated CASSANDRA-924:
-------------------------------

    Attachment:     (was: 924.patch)

> lost data replica not restored after server restart
> ---------------------------------------------------
>
>                 Key: CASSANDRA-924
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-924
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5, 0.6, 0.6.1, 0.7
>         Environment: CentOS 5.2
>            Reporter: Jianing hu
>             Fix For: 0.5, 0.6, 0.6.1, 0.7
>
>         Attachments: 924-0.5.patch, 924.patch, cs1.log, cs1.log, cs2.log, cs2.log, cs3.log, cs3.log, error.log, error.log, error.log, storage-conf.xml
>
>
> With Replicationfactor=2, if a server is brought down and its data directory wiped out, it doesn't restore its data replica after restart and nodeprobe repair.
> Steps to reproduce:
> 1) Bring up a cluster with three servers cs1,2,3, with their initial token set to 'foo3', 'foo6', and 'foo9', respectively. ReplicationFactor is set to 2 on all 3.
> 2) Insert 9 columns with keys from 'foo1' to 'foo9', and flush. Now I have foo1,2,3,7,8,9 on cs1, foo1,2,3,4,5,6, on cs2, and foo4,5,6,7,8,9
> on cs3. So far so good
> 3) Bring down cs3 and wipe out its data directory
> 4) Bring up cs3
> 5) run nodeprobe repair Keyspace1 on cs3, the flush
> At this point I expect to see cs3 getting its data back. But there's nothing in its data directory. I also tried getting all columns with
> ConsistencyLevel::ALL to see if that'll do a read pair. But still cs3's data directory is empty.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.