You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Duncan Sands (JIRA)" <ji...@apache.org> on 2014/08/31 22:13:22 UTC
[jira] [Commented] (CASSANDRA-6651) Repair hanging
[ https://issues.apache.org/jira/browse/CASSANDRA-6651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116877#comment-14116877 ]
Duncan Sands commented on CASSANDRA-6651:
-----------------------------------------
I'm on 2.0.10, and seem to have hit this while running repair this weekend.
The repair #e3b1fc60-3010-11e4-bd56-390059926170 on node 192.168.21.13 got stuck (see log snippet below). Summary of the log snippet: it requested a merkle tree for "trades" to nodes 172.18.68.138 and 192.168.21.13; it almost immediately got a merkle tree back from 192.168.21.13 (itself!) but never got anything back from 172.18.68.138. The log snippet:
INFO [AntiEntropyStage:1] 2014-08-30 08:43:22,823 Validator.java (line 254) [repair #d1667400-3010-11e4-847a-71045e056b2b] Sending completed merkle tree to /192.168.60.141 for tick_data/historical_prices
INFO [AntiEntropyStage:1] 2014-08-30 08:43:25,382 Validator.java (line 254) [repair #e0c033a0-3010-11e4-b654-51c077eaf311] Sending completed merkle tree to /172.18.68.138 for all_production/table_metadata
INFO [AntiEntropyStage:1] 2014-08-30 08:43:25,517 RepairSession.java (line 166) [repair #e3b1fc60-3010-11e4-bd56-390059926170] Received merkle tree for swxess_connections from /172.18.68.138
INFO [AntiEntropyStage:1] 2014-08-30 08:43:25,517 RepairJob.java (line 143) [repair #e3b1fc60-3010-11e4-bd56-390059926170] requesting merkle trees for trades (to [/172.18.68.138, /192.168.21.13])
INFO [RepairJobTask:1] 2014-08-30 08:43:25,521 Differencer.java (line 67) [repair #e3b1fc60-3010-11e4-bd56-390059926170] Endpoints /192.168.21.13 and /172.18.68.138 are consistent for swxess_connections
INFO [AntiEntropyStage:1] 2014-08-30 08:43:25,521 RepairSession.java (line 223) [repair #e3b1fc60-3010-11e4-bd56-390059926170] swxess_connections is fully synced
INFO [AntiEntropyStage:1] 2014-08-30 08:43:25,551 RepairSession.java (line 166) [repair #e3b1fc60-3010-11e4-bd56-390059926170] Received merkle tree for trades from /192.168.21.13
It is now 36 hours later and it never got a merkle tree back from 172.18.68.138.
In fact 172.18.68.138 shows no sign of having received the merkle tree request for "trades". Log snippet from around the time of the request:
INFO [AntiEntropyStage:1] 2014-08-30 08:43:25,372 Validator.java (line 254) [repair #e3b1fc60-3010-11e4-bd56-390059926170] Sending completed merkle tree to /192.168.21.13 for zrh_simulation/swxess_connections
INFO [AntiEntropyStage:1] 2014-08-30 08:43:25,385 RepairSession.java (line 166) [repair #e0c033a0-3010-11e4-b65451c077eaf311] Received merkle tree for table_metadata from /172.18.68.139
INFO [AntiEntropyStage:1] 2014-08-30 08:43:25,401 RepairSession.java (line 166) [repair #e0c033a0-3010-11e4-b65451c077eaf311] Received merkle tree for table_metadata from /172.18.68.138
INFO [AntiEntropyStage:1] 2014-08-30 08:43:25,817 Validator.java (line 254) [repair #ef7cc8e0-3010-11e4-afa2-87e1f9d131a9] Sending completed merkle tree to /192.168.60.142 for OpsCenter/rollups300
INFO [AntiEntropyStage:1] 2014-08-30 08:43:26,980 Validator.java (line 254) [repair #ef7cc8e0-3010-11e4-afa2-87e1f9d131a9] Sending completed merkle tree to /192.168.60.142 for OpsCenter/rollups7200
INFO [STREAM-INIT-/192.168.60.142:43638] 2014-08-30 08:43:27,007 StreamResultFuture.java (line 121) [Stream #f23afe81-3010-11e4-afa2-87e1f9d131a9] Received streaming plan for Repair
INFO [STREAM-INIT-/192.168.60.141:33019] 2014-08-30 08:43:27,012 StreamResultFuture.java (line 121) [Stream #f23afe80-3010-11e4-847a-71045e056b2b] Received streaming plan for Repair
INFO [STREAM-IN-/192.168.60.142] 2014-08-30 08:43:27,028 StreamResultFuture.java (line 173) [Stream #f23afe81-3010-11e4-afa2-87e1f9d131a9] Prepare completed. Receiving 4 files(543791 bytes), sending 4 files(570871 bytes)
INFO [STREAM-IN-/192.168.60.141] 2014-08-30 08:43:27,032 StreamResultFuture.java (line 173) [Stream #f23afe80-3010-11e4-847a-71045e056b2b] Prepare completed. Receiving 5 files(569602 bytes), sending 4 files(570871 bytes)
The request doesn't turn up anywhere in the log files for the following 36 hours.
This seems to be the same issue as discussed in this ticket.
> Repair hanging
> --------------
>
> Key: CASSANDRA-6651
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6651
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Eitan Eibschutz
> Assignee: Yuki Morishita
>
> Hi,
> We have a 12 node cluster in PROD environment and we've noticed that repairs are never finishing. The behavior that we've observed is that a repair process will run until at some point it hangs and no other processing is happening.
> For example, at the moment, I have a repair process that has been running for two days and not finishing:
> nodetool tpstats is showing 2 active and 2 pending AntiEntropySessions
> nodetool compactionstats is showing:
> pending tasks: 0
> Active compaction remaining time : n/a
> nodetools netstats is showing:
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 142110
> Mismatch (Background): 0
> Pool Name Active Pending Completed
> Commands n/a 0 107589657
> Responses n/a 0 116430785
> The last entry that I see in the log is:
> INFO [AntiEntropySessions:18] 2014-02-03 04:01:39,145 RepairJob.java (line 116) [repair #ae78c6c0-8c2b-11e3-b950-c3b81a36bc9b] requesting merkle trees for MyCF (to [/x.x.x.x, /y.y.y.y, /z.z.z.z])
> The repair started at 4am so it stopped after 1:40 minute.
> On node y.y.y.y I can see this in the log:
> INFO [MiscStage:1] 2014-02-03 04:01:38,985 ColumnFamilyStore.java (line 740) Enqueuing flush of Memtable-MyCF@1290890489(2176/5931 serialized/live bytes, 32 ops)
> INFO [FlushWriter:411] 2014-02-03 04:01:38,986 Memtable.java (line 333) Writing Memtable-MyCF@1290890489(2176/5931 serialized/live bytes, 32 ops)
> INFO [FlushWriter:411] 2014-02-03 04:01:39,048 Memtable.java (line 373) Completed flushing /var/lib/cassandra/main-db/data/MyKS/MyCF/MyKS-MyCF-jb-518-Data.db (1789 bytes) for commitlog position ReplayPosition(segmentId=1390437013339, position=21868792)
> INFO [ScheduledTasks:1] 2014-02-03 05:00:04,794 ColumnFamilyStore.java (line 740) Enqueuing flush of Memtable-compaction_history@1649414699(1635/17360 serialized/live bytes, 42 ops)
> So for some reason the merkle tree for this CF is never sent back to the node being repaired and it's hanging.
> I've also noticed that sometimes, restarting node y.y.y.y will cause the repair to resume.
> Another observation is that sometimes when restarting y.y.y.y it will not start with these errors:
> ERROR 16:34:18,485 Exception encountered during startup
> java.lang.IllegalStateException: Unfinished compactions reference missing sstables. This should never happen since compactions are marked finished before we start removing the old sstables.
> at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:495)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:264)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
> java.lang.IllegalStateException: Unfinished compactions reference missing sstables. This should never happen since compactions are marked finished before we start removing the old sstables.
> at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:495)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:264)
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:461)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504)
> Exception encountered during startup: Unfinished compactions reference missing sstables. This should never happen since compactions are marked finished before we start removing the old sstables.
> And it will only restart after manually cleaning the compactions_in-progress folder.
> I'm not sure if these two issues are related but we've seen both on all the nodes in our cluster.
> I'll be happy to provide more info if needed as we are not sure what could cause this behavior.
> Another thing in our environment is that some of the Cassandra nodes have more than one network interface and RPC is listening on 0.0.0.0, not sure if it has anything to do with this.
> Thanks,
> Eitan
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)