You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Razi Khaja (JIRA)" <ji...@apache.org> on 2014/09/12 23:31:34 UTC
[jira] [Comment Edited] (CASSANDRA-7904) Repair hangs
[ https://issues.apache.org/jira/browse/CASSANDRA-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132122#comment-14132122 ]
Razi Khaja edited comment on CASSANDRA-7904 at 9/12/14 9:30 PM:
----------------------------------------------------------------
We have 3 data centers, each with 4 nodes (running on physical machines not on EC2). We have been running Cassandra 2.0.6 and have not been able to successfully run *nodetool repair* on any of our nodes (except when no data or almost no data was loaded into our keyspaces). We upgraded to Cassandra 2.0.10 hoping that this issue of *Lost notification* during the *nodetool repair* would be fixed, but as you can see from the log below, we still have not been able to successfully run *nodetool repair*.
{code}
[2014-09-12 11:08:02,131] Nothing to repair for keyspace 'system'
[2014-09-12 11:08:02,179] Starting repair command #10, repairing 1389 ranges for keyspace megalink
[2014-09-12 11:12:02,196] Lost notification. You should check server log for repair status of keyspace megalink
[2014-09-12 11:12:02,258] Starting repair command #11, repairing 1389 ranges for keyspace megalink_dev
[2014-09-12 11:12:02,331] Repair command #11 finished
[2014-09-12 11:12:02,346] Starting repair command #12, repairing 512 ranges for keyspace system_traces
[2014-09-12 11:13:02,349] Lost notification. You should check server log for repair status of keyspace system_traces
{code}
If there any more details that are needed to help solve this problem please let me know and I will do my best to provide them. Please let me know how I can help.
was (Author: razi.khaja@gmail.com):
We have 3 data centers, each with 4 nodes (running on physical machines not on EC2). We have been running Cassandra 2.0.6 and have not been able to successfully run *nodetool repair* on any of our nodes (except when no data or almost no data was loaded into our keyspaces). We upgraded to Cassandra 2.0.10 hoping that this issue of *Lost notification* during the *nodetool repair* would be fixed, but as you can see from the log below, we still have not been able to successfully run *nodetool repair*.
{code}
[2014-09-12 11:08:02,131] Nothing to repair for keyspace 'system'
[2014-09-12 11:08:02,179] Starting repair command #10, repairing 1389 ranges for keyspace megalink
[2014-09-12 11:12:02,196] Lost notification. You should check server log for repair status of keyspace megalink
[2014-09-12 11:12:02,258] Starting repair command #11, repairing 1389 ranges for keyspace megalink_dev
[2014-09-12 11:12:02,331] Repair command #11 finished
[2014-09-12 11:12:02,346] Starting repair command #12, repairing 512 ranges for keyspace system_traces
[2014-09-12 11:13:02,349] Lost notification. You should check server log for repair status of keyspace system_traces
{code}
If there any more details that are needed to help solve this problem please let me know and I will do my best to provide them.
> Repair hangs
> ------------
>
> Key: CASSANDRA-7904
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7904
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: C* 2.0.10, ubuntu 14.04, Java HotSpot(TM) 64-Bit Server, java version "1.7.0_45"
> Reporter: Duncan Sands
> Attachments: ls-172.18.68.138, ls-192.168.21.13, ls-192.168.60.134, ls-192.168.60.136
>
>
> Cluster of 22 nodes spread over 4 data centres. Not used on the weekend, so repair is run on all nodes (in a staggered fashion) on the weekend. Nodetool options: -par -pr. There is usually some overlap in the repairs: repair on one node may well still be running when repair is started on the next node. Repair hangs for some of the nodes almost every weekend. It hung last weekend, here are the details:
> In the whole cluster, only one node had an exception since C* was last restarted. This node is 192.168.60.136 and the exception is harmless: a client disconnected abruptly.
> tpstats
> 4 nodes have a non-zero value for "active" or "pending" in AntiEntropySessions. These nodes all have Active => 1 and Pending => 1. The nodes are:
> 192.168.21.13 (data centre R)
> 192.168.60.134 (data centre A)
> 192.168.60.136 (data centre A)
> 172.18.68.138 (data centre Z)
> compactionstats:
> No compactions. All nodes have:
> pending tasks: 0
> Active compaction remaining time : n/a
> netstats:
> All except one node have nothing. One node (192.168.60.131, not one of the nodes listed in the tpstats section above) has (note the Responses Pending value of 1):
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 4233
> Mismatch (Blocking): 0
> Mismatch (Background): 243
> Pool Name Active Pending Completed
> Commands n/a 0 34785445
> Responses n/a 1 38567167
> Repair sessions
> I looked for repair sessions that failed to complete. On 3 of the 4 nodes mentioned in tpstats above I found that they had sent merkle tree requests and got responses from all but one node. In the log file for the node that failed to respond there is no sign that it ever received the request. On 1 node (172.18.68.138) it looks like responses were received from every node, some streaming was done, and then... nothing. Details:
> Node 192.168.21.13 (data centre R):
> Sent merkle trees to /172.18.33.24, /192.168.60.140, /192.168.60.142, /172.18.68.139, /172.18.68.138, /172.18.33.22, /192.168.21.13 for table brokers, never got a response from /172.18.68.139. On /172.18.68.139, just before this time it sent a response for the same repair session but a different table, and there is no record of it receiving a request for table brokers.
> Node 192.168.60.134 (data centre A):
> Sent merkle trees to /172.18.68.139, /172.18.68.138, /192.168.60.132, /192.168.21.14, /192.168.60.134 for table swxess_outbound, never got a response from /172.18.68.138. On /172.18.68.138, just before this time it sent a response for the same repair session but a different table, and there is no record of it receiving a request for table swxess_outbound.
> Node 192.168.60.136 (data centre A):
> Sent merkle trees to /192.168.60.142, /172.18.68.139, /192.168.60.136 for table rollups7200, never got a response from /172.18.68.139. This repair session is never mentioned in the /172.18.68.139 log.
> Node 172.18.68.138 (data centre Z):
> The issue here seems to be repair session #a55c16e1-35eb-11e4-8e7e-51c077eaf311. It got responses for all its merkle tree requests, did some streaming, but seems to have stopped after finishing with one table (rollups60). I found it as follows: it is the only repair for which there is no "session completed successfully" message in the log.
> Some log file snippets are attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)