You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Richard Low (JIRA)" <ji...@apache.org> on 2011/07/25 19:14:09 UTC
[jira] [Created] (CASSANDRA-2947) New nodes always think dead nodes
are alive
New nodes always think dead nodes are alive
-------------------------------------------
Key: CASSANDRA-2947
URL: https://issues.apache.org/jira/browse/CASSANDRA-2947
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Richard Low
If a new node is brought up while a node is down, it will think it is up forever.
To reproduce:
Take nodes A, B and C.
1. Bring up nodes A and B in a cluster
2. Take down B and wait for A to mark it as down
3. Bring up C with A as a seed
4. nodetool ring on C shows all 3 nodes as up and never marks B as down
The problem is that the failure detector never learns about node B - FD.report is never called for B. This means requests are constantly routed to B from C and timeout, but they should fail with UnavailableException.
The attached (hack) patch appears to fix it, but I expect the problem is actually elsewhere in the gossip code.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2947) New nodes always think dead nodes
are alive
Posted by "Richard Low (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Low updated CASSANDRA-2947:
-----------------------------------
Attachment: 2947.txt
> New nodes always think dead nodes are alive
> -------------------------------------------
>
> Key: CASSANDRA-2947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2947
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.2
> Reporter: Richard Low
> Attachments: 2947.txt
>
>
> If a new node is brought up while a node is down, it will think it is up forever.
> To reproduce:
> Take nodes A, B and C.
> 1. Bring up nodes A and B in a cluster
> 2. Take down B and wait for A to mark it as down
> 3. Bring up C with A as a seed
> 4. nodetool ring on C shows all 3 nodes as up and never marks B as down
> The problem is that the failure detector never learns about node B - FD.report is never called for B. This means requests are constantly routed to B from C and timeout, but they should fail with UnavailableException.
> The attached (hack) patch appears to fix it, but I expect the problem is actually elsewhere in the gossip code.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2947) New nodes always think dead
nodes are alive
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-2947:
-------------------------------------------
Assignee: Brandon Williams
> New nodes always think dead nodes are alive
> -------------------------------------------
>
> Key: CASSANDRA-2947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2947
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.2
> Reporter: Richard Low
> Assignee: Brandon Williams
> Attachments: 2947.txt
>
>
> If a new node is brought up while a node is down, it will think it is up forever.
> To reproduce:
> Take nodes A, B and C.
> 1. Bring up nodes A and B in a cluster
> 2. Take down B and wait for A to mark it as down
> 3. Bring up C with A as a seed
> 4. nodetool ring on C shows all 3 nodes as up and never marks B as down
> The problem is that the failure detector never learns about node B - FD.report is never called for B. This means requests are constantly routed to B from C and timeout, but they should fail with UnavailableException.
> The attached (hack) patch appears to fix it, but I expect the problem is actually elsewhere in the gossip code.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2947) New nodes always think dead
nodes are alive
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams resolved CASSANDRA-2947.
-----------------------------------------
Resolution: Duplicate
I tested against the 0.8 branch and it doesn't reproduce. I suspect something I did in CASSANDRA-2496 resolved it.
{noformat}
TRACE 20:20:12,535 Performing status check ...
TRACE 20:20:12,535 PHI for /10.179.111.137 : 8.60511086443103
TRACE 20:20:12,535 marking as dead /10.179.111.137
INFO 20:20:12,536 InetAddress /10.179.111.137 is now dead.
{noformat}
> New nodes always think dead nodes are alive
> -------------------------------------------
>
> Key: CASSANDRA-2947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2947
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.2
> Reporter: Richard Low
> Assignee: Brandon Williams
> Attachments: 2947.txt
>
>
> If a new node is brought up while a node is down, it will think it is up forever.
> To reproduce:
> Take nodes A, B and C.
> 1. Bring up nodes A and B in a cluster
> 2. Take down B and wait for A to mark it as down
> 3. Bring up C with A as a seed
> 4. nodetool ring on C shows all 3 nodes as up and never marks B as down
> The problem is that the failure detector never learns about node B - FD.report is never called for B. This means requests are constantly routed to B from C and timeout, but they should fail with UnavailableException.
> The attached (hack) patch appears to fix it, but I expect the problem is actually elsewhere in the gossip code.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2947) New nodes always think dead
nodes are alive
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070726#comment-13070726 ]
Brandon Williams commented on CASSANDRA-2947:
---------------------------------------------
bq. The problem is that the failure detector never learns about node B - FD.report is never called for B.
This isn't quite right, the FD knows about B and is still calculating phi for it, but it is never reported for some reason:
{noformat}
TRACE 20:12:12,201 Performing status check ...
TRACE 20:12:12,201 PHI for /10.179.111.137 : 19.923693651793577
TRACE 20:12:12,201 PHI for /10.179.65.102 : 0.43267044519934433
{noformat}
> New nodes always think dead nodes are alive
> -------------------------------------------
>
> Key: CASSANDRA-2947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2947
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.2
> Reporter: Richard Low
> Attachments: 2947.txt
>
>
> If a new node is brought up while a node is down, it will think it is up forever.
> To reproduce:
> Take nodes A, B and C.
> 1. Bring up nodes A and B in a cluster
> 2. Take down B and wait for A to mark it as down
> 3. Bring up C with A as a seed
> 4. nodetool ring on C shows all 3 nodes as up and never marks B as down
> The problem is that the failure detector never learns about node B - FD.report is never called for B. This means requests are constantly routed to B from C and timeout, but they should fail with UnavailableException.
> The attached (hack) patch appears to fix it, but I expect the problem is actually elsewhere in the gossip code.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira