You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Greg Doermann (JIRA)" <ji...@apache.org> on 2017/02/27 17:06:45 UTC
[jira] [Comment Edited] (CASSANDRA-13042) The two cassandra nodes
suddenly encounter hints each other and failed replaying.
[ https://issues.apache.org/jira/browse/CASSANDRA-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886128#comment-15886128 ]
Greg Doermann edited comment on CASSANDRA-13042 at 2/27/17 5:05 PM:
--------------------------------------------------------------------
We are getting a constant loop of restarts on the affected servers. It tries to send the server the hints and ends up restarting when all of them fail:
{quote}
INFO [HintedHandoff:5] 2017-02-27 16:52:13,959 HintedHandOffManager.java:367 - Started hinted handoff for host: 00000000-0000-0000-000-000000000000 with IP: /1.1.1.1
ERROR [HintedHandoff:10] 2017-02-27 16:52:13,961 CassandraDaemon.java:185 - Exception in thread Thread[HintedHandoff:10,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.net.WriteCallbackInfo.<init>(WriteCallbackInfo.java:49) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:639) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:703) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:474) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:354) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:93) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:566) ~[apache-cassandra-2.2.5.jar:2.2.5]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74]
INFO [HintedHandoff:5] 2017-02-27 16:52:19,465 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.1; aborting (523 delivered), error : Operation timed out - received only 0 responses.
{quote}
was (Author: gdoermann):
We are getting a constant loop of restarts on the affected servers. It tries to send the server the hints and ends up restarting when all of them fail:
```INFO [HintedHandoff:5] 2017-02-27 16:52:13,959 HintedHandOffManager.java:367 - Started hinted handoff for host: 00000000-0000-0000-000-000000000000 with IP: /1.1.1.1
ERROR [HintedHandoff:10] 2017-02-27 16:52:13,961 CassandraDaemon.java:185 - Exception in thread Thread[HintedHandoff:10,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.net.WriteCallbackInfo.<init>(WriteCallbackInfo.java:49) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:639) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:703) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:474) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:354) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:93) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:566) ~[apache-cassandra-2.2.5.jar:2.2.5]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74]
INFO [HintedHandoff:5] 2017-02-27 16:52:19,465 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.1; aborting (523 delivered), error : Operation timed out - received only 0 responses.
```
> The two cassandra nodes suddenly encounter hints each other and failed replaying.
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-13042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13042
> Project: Cassandra
> Issue Type: Bug
> Reporter: YheonHo.Choi
> Priority: Critical
> Attachments: out_2.2.2.1.txt, out_2.2.2.2.txt
>
>
> Although there are no changes to cassandra, two node suddenly encounter hints and failed replaying.
> Any commands like disablethrift, disablegossip can not solve the above problem and the only way was restart.
> When we check the status of cluster, all nodes are looks UN but describecluster show unreachable each other.
> Here's the state of the cassandra during the above problem occurred.
> IP addresses in report anonymized:
> cassandra version: 2.2.5
> node 1 = 1.1.1.1
> node 2 = 1.1.1.2
> others = x.x.x.x
> system.log
> {code}
> ## result of nodetool status on 1.1.1.1
> INFO [HintedHandoff:1] 2016-11-24 06:15:07,969 HintedHandOffManager.java:367 - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with IP: /1.1.1.2
> INFO [HintedHandoff:1] 2016-11-24 06:15:09,969 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> INFO [HintedHandoff:2] 2016-11-24 06:25:09,736 HintedHandOffManager.java:367 - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with IP: /1.1.1.2
> INFO [HintedHandoff:2] 2016-11-24 06:25:11,738 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> WARN [MemtableFlushWriter:55270] 2016-11-24 06:25:12,625 BigTableWriter.java:184 - Writing large partition system/hints:d640677d-f354-aa8c-be89-d2a1648c24b2 (109029803 bytes)
> WARN [CompactionExecutor:37908] 2016-11-24 06:35:23,682 BigTableWriter.java:184 - Writing large partition system/hints:8caa54f3-7d67-40d6-b224-8c64a1d289be (250651758 bytes)
> INFO [HintedHandoff:1] 2016-11-24 06:35:23,727 HintedHandOffManager.java:367 - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with IP: /1.1.1.2
> INFO [HintedHandoff:1] 2016-11-24 06:35:25,728 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> WARN [CompactionExecutor:37909] 2016-11-24 06:45:53,615 BigTableWriter.java:184 - Writing large partition system/hints:8caa54f3-7d67-40d6-b224-8c64a1d289be (340801514 bytes)
> INFO [HintedHandoff:2] 2016-11-24 06:45:53,718 HintedHandOffManager.java:367 - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with IP: /1.1.1.2
> INFO [HintedHandoff:2] 2016-11-24 06:45:55,719 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> WARN [CompactionExecutor:37912] 2016-11-24 06:56:20,884 BigTableWriter.java:184 - Writing large partition system/hints:8caa54f3-7d67-40d6-b224-8c64a1d289be (472465093 bytes)
> INFO [HintedHandoff:1] 2016-11-24 06:56:20,966 HintedHandOffManager.java:367 - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with IP: /1.1.1.2
> INFO [HintedHandoff:1] 2016-11-24 06:56:22,967 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> WARN [CompactionExecutor:37911] 2016-11-24 07:07:12,568 BigTableWriter.java:184 - Writing large partition system/hints:8caa54f3-7d67-40d6-b224-8c64a1d289be (577392172 bytes)
> INFO [HintedHandoff:2] 2016-11-24 07:07:12,643 HintedHandOffManager.java:367 - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with IP: /1.1.1.2
> INFO [HintedHandoff:2] 2016-11-24 07:07:14,643 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> INFO [IndexSummaryManager:1] 2016-11-24 07:09:15,929 IndexSummaryRedistribution.java:74 - Redistributing index summaries
> ## result of nodetool status on 1.1.1.2
> INFO [HintedHandoff:1] 2016-11-24 06:11:37,300 HintedHandOffManager.java:367 - Started hinted handoff for host: b79124e9-394c-4400-a8e7-a0c94aec6878 with IP: /1.1.1.1
> INFO [HintedHandoff:1] 2016-11-24 06:11:39,301 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.1; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> INFO [HintedHandoff:2] 2016-11-24 06:22:17,946 HintedHandOffManager.java:367 - Started hinted handoff for host: b79124e9-394c-4400-a8e7-a0c94aec6878 with IP: /1.1.1.1
> INFO [HintedHandoff:2] 2016-11-24 06:22:19,948 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.1; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> INFO [IndexSummaryManager:1] 2016-11-24 06:27:00,177 IndexSummaryRedistribution.java:74 - Redistributing index summaries
> WARN [CompactionExecutor:2315] 2016-11-24 06:32:28,159 BigTableWriter.java:184 - Writing large partition system/hints:b79124e9-394c-4400-a8e7-a0c94aec6878 (121683824 bytes)
> INFO [HintedHandoff:1] 2016-11-24 06:32:28,338 HintedHandOffManager.java:367 - Started hinted handoff for host: b79124e9-394c-4400-a8e7-a0c94aec6878 with IP: /1.1.1.1
> INFO [HintedHandoff:1] 2016-11-24 06:32:30,340 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.1; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> WARN [MemtableFlushWriter:8728] 2016-11-24 06:32:31,272 BigTableWriter.java:184 - Writing large partition system/hints:00444c39-e924-91b7-7868-ec4ac9a0e7a8 (108691834 bytes)
> WARN [CompactionExecutor:2316] 2016-11-24 06:42:58,591 BigTableWriter.java:184 - Writing large partition system/hints:b79124e9-394c-4400-a8e7-a0c94aec6878 (327308871 bytes)
> INFO [HintedHandoff:2] 2016-11-24 06:42:58,661 HintedHandOffManager.java:367 - Started hinted handoff for host: b79124e9-394c-4400-a8e7-a0c94aec6878 with IP: /1.1.1.1
> INFO [HintedHandoff:2] 2016-11-24 06:43:00,662 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.1; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> WARN [MemtableFlushWriter:8739] 2016-11-24 06:52:42,921 BigTableWriter.java:184 - Writing large partition system/hints:00444c39-e924-91b7-7868-ec4ac9a0e7a8 (105086190 bytes)
> WARN [CompactionExecutor:2316] 2016-11-24 06:53:43,722 BigTableWriter.java:184 - Writing large partition system/hints:b79124e9-394c-4400-a8e7-a0c94aec6878 (449333141 bytes)
> INFO [HintedHandoff:1] 2016-11-24 06:53:43,769 HintedHandOffManager.java:367 - Started hinted handoff for host: b79124e9-394c-4400-a8e7-a0c94aec6878 with IP: /1.1.1.1
> INFO [HintedHandoff:1] 2016-11-24 06:53:45,770 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.1; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> WARN [CompactionExecutor:2316] 2016-11-24 07:04:29,379 BigTableWriter.java:184 - Writing large partition system/hints:b79124e9-394c-4400-a8e7-a0c94aec6878 (460781059 bytes)
> INFO [HintedHandoff:2] 2016-11-24 07:04:29,572 HintedHandOffManager.java:367 - Started hinted handoff for host: b79124e9-394c-4400-a8e7-a0c94aec6878 with IP: /1.1.1.1
> INFO [HintedHandoff:2] 2016-11-24 07:04:31,574 HintedHandOffManager.java:486 - Failed replaying hints to /1.1.1.1; aborting (0 delivered), error : Operation timed out - received only 0 responses.
> WARN [MemtableFlushWriter:8749] 2016-11-24 07:04:32,553 BigTableWriter.java:184 - Writing large partition system/hints:00444c39-e924-91b7-7868-ec4ac9a0e7a8 (117146652 bytes)
> {code}
> nodetool status
> {code}
> ## result of nodetool status on 1.1.1.1
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID Rack
> UN 1.1.1.2 252.48 GB 256 ? 8caa54f3-7d67-40d6-b224-8c64a1d289be rack1
> UN x.x.x.x 316.6 GB 256 ? a08fbe58-779f-49f7-a0be-0a288d18c059 rack1
> UN x.x.x.x 240.92 GB 256 ? d2cf2695-25b7-4aad-92ab-9a0b593df5fb rack1
> UN x.x.x.x 242.68 GB 256 ? a01d3304-5d8d-4ac9-935f-597ebf038e70 rack1
> UN x.x.x.x 326.76 GB 256 ? b79124e9-394c-4400-a8e7-a0c94aec6878 rack1
> UN x.x.x.x 296.61 GB 256 ? 0ec779f9-3ee4-41a2-80b2-f27f25521964 rack1
> UN x.x.x.x 309.81 GB 256 ? 8e6417ec-e375-40a4-bb8d-3d99148846ff rack1
> UN x.x.x.x 312.28 GB 256 ? b39ee3e9-8f69-4f5c-8473-0e906c2301ca rack1
> UN x.x.x.x 299.9 GB 256 ? 0ab413a9-5608-4d7b-97dc-6ed655e76dd0 rack1
> UN x.x.x.x 241.81 GB 256 ? 81d4608f-d1df-4f52-b104-5f1c7a3b4c96 rack1
> UN x.x.x.x 228.76 GB 256 ? afc0ae8e-1126-426f-aa5b-0fce5578531c rack1
> UN x.x.x.x 301.13 GB 256 ? 62323ea7-8d66-46fd-93e1-ff9bbf772c17 rack1
> UN x.x.x.x 307.39 GB 256 ? 7939c565-dc6e-43ad-b6ed-cfa3a19b20ab rack1
> UN x.x.x.x 259.21 GB 256 ? afcdc0a9-de5a-446d-868e-ee1dabfc0107 rack1
> UN x.x.x.x 294.16 GB 256 ? 5582b905-7136-4e87-b1a9-d450ea2eb304 rack1
> UN x.x.x.x 285.28 GB 256 ? 6dabba2e-085d-497b-a97e-b6ebf65258d6 rack1
> UN x.x.x.x 305.59 GB 256 ? 10b51458-800b-46f2-a80a-c2f2bbe1e46d rack1
> UN x.x.x.x 317.64 GB 256 ? 79a69fe9-de49-49d3-b8cb-fbc59e08a821 rack1
> UN x.x.x.x 303.85 GB 256 ? 71db8e6a-cdbb-4924-860c-34e40731719f rack1
> Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
> ## result of nodetool status on 1.1.1.2
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID Rack
> UN x.x.x.x 252.48 GB 256 ? 8caa54f3-7d67-40d6-b224-8c64a1d289be rack1
> UN x.x.x.x 316.6 GB 256 ? a08fbe58-779f-49f7-a0be-0a288d18c059 rack1
> UN x.x.x.x 240.92 GB 256 ? d2cf2695-25b7-4aad-92ab-9a0b593df5fb rack1
> UN x.x.x.x 242.68 GB 256 ? a01d3304-5d8d-4ac9-935f-597ebf038e70 rack1
> UN 1.1.1.1 326.76 GB 256 ? b79124e9-394c-4400-a8e7-a0c94aec6878 rack1
> UN x.x.x.x 296.61 GB 256 ? 0ec779f9-3ee4-41a2-80b2-f27f25521964 rack1
> UN x.x.x.x 309.81 GB 256 ? 8e6417ec-e375-40a4-bb8d-3d99148846ff rack1
> UN x.x.x.x 312.28 GB 256 ? b39ee3e9-8f69-4f5c-8473-0e906c2301ca rack1
> UN x.x.x.x 299.9 GB 256 ? 0ab413a9-5608-4d7b-97dc-6ed655e76dd0 rack1
> UN x.x.x.x 241.81 GB 256 ? 81d4608f-d1df-4f52-b104-5f1c7a3b4c96 rack1
> UN x.x.x.x 228.76 GB 256 ? afc0ae8e-1126-426f-aa5b-0fce5578531c rack1
> UN x.x.x.x 301.13 GB 256 ? 62323ea7-8d66-46fd-93e1-ff9bbf772c17 rack1
> UN x.x.x.x 307.39 GB 256 ? 7939c565-dc6e-43ad-b6ed-cfa3a19b20ab rack1
> UN x.x.x.x 259.21 GB 256 ? afcdc0a9-de5a-446d-868e-ee1dabfc0107 rack1
> UN x.x.x.x 294.16 GB 256 ? 5582b905-7136-4e87-b1a9-d450ea2eb304 rack1
> UN x.x.x.x 285.28 GB 256 ? 6dabba2e-085d-497b-a97e-b6ebf65258d6 rack1
> UN x.x.x.x 305.59 GB 256 ? 10b51458-800b-46f2-a80a-c2f2bbe1e46d rack1
> UN x.x.x.x 317.64 GB 256 ? 79a69fe9-de49-49d3-b8cb-fbc59e08a821 rack1
> UN x.x.x.x 303.85 GB 256 ? 71db8e6a-cdbb-4924-860c-34e40731719f rack1
> Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
> {code}
> nodetool describecluster
> {code}
> ## result of nodetool describecluster on 1.1.1.1
> Cluster Information:
> Name: metric
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9: [x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x]
> UNREACHABLE: [1.1.1.2]
> ## result of nodetool describecluster on 1.1.1.2
> Cluster Information:
> Name: metric
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9: [x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x]
> UNREACHABLE: [1.1.1.1]
> {code}
> We've looked at whether schema is different.
> nodetool gossipinfo
> {code}
> ## result of nodetool gossipinfo on 1.1.1.1
> /x.x.x.x
> generation:1464511828
> heartbeat:46798705
> STATUS:14:NORMAL,-1022166408914492069
> LOAD:46798599:3.40042922062E11
> SCHEMA:37196731:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:46798704:0.0
> NET_VERSION:1:9
> HOST_ID:2:a08fbe58-779f-49f7-a0be-0a288d18c059
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /1.1.1.1
> generation:1471493280
> heartbeat:25624228
> STATUS:25498:NORMAL,-1023351769271276256
> LOAD:25624146:3.51052126949E11
> SCHEMA:16021810:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:1.1.1.1
> SEVERITY:25624230:0.0
> NET_VERSION:1:9
> HOST_ID:2:b79124e9-394c-4400-a8e7-a0c94aec6878
> RPC_READY:25524:true
> TOKENS:25497:<hidden>
> /x.x.x.x
> generation:1467205802
> heartbeat:38628440
> STATUS:14:NORMAL,-1124164164231176551
> LOAD:38628359:3.06404316564E11
> SCHEMA:29026044:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:38628439:0.0
> NET_VERSION:1:9
> HOST_ID:2:6dabba2e-085d-497b-a97e-b6ebf65258d6
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1479090215
> heartbeat:2581923
> STATUS:16:NORMAL,-1034930499164644146
> LOAD:2581842:2.60571265374E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:2581922:0.0
> NET_VERSION:1:9
> HOST_ID:2:a01d3304-5d8d-4ac9-935f-597ebf038e70
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1479089630
> heartbeat:2583693
> STATUS:16:NORMAL,-1104754836621297034
> LOAD:2583656:2.59767873976E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:2583692:0.0
> NET_VERSION:1:9
> HOST_ID:2:81d4608f-d1df-4f52-b104-5f1c7a3b4c96
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1471482865
> heartbeat:25655714
> STATUS:28484:NORMAL,-1005894856009564957
> LOAD:25655674:3.35493584628E11
> SCHEMA:16053376:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:25655716:0.0
> NET_VERSION:1:9
> HOST_ID:2:b39ee3e9-8f69-4f5c-8473-0e906c2301ca
> RPC_READY:28512:true
> TOKENS:28483:<hidden>
> /1.1.1.2
> generation:1479090411
> heartbeat:2581314
> STATUS:14:NORMAL,-1016248794247274800
> LOAD:2581277:2.71127001342E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:1.1.1.2
> SEVERITY:2581313:0.0
> NET_VERSION:1:9
> HOST_ID:2:8caa54f3-7d67-40d6-b224-8c64a1d289be
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1465163903
> heartbeat:44821598
> STATUS:14:NORMAL,-1077736070944113862
> LOAD:44821597:3.26381003723E11
> SCHEMA:35219220:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:44821595:0.0
> NET_VERSION:1:9
> HOST_ID:2:71db8e6a-cdbb-4924-860c-34e40731719f
> RPC_READY:42:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1467203191
> heartbeat:38636328
> STATUS:14:NORMAL,-100689147463310850
> LOAD:38636153:3.15710669743E11
> SCHEMA:29033962:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:38636327:0.0
> NET_VERSION:1:9
> HOST_ID:2:5582b905-7136-4e87-b1a9-d450ea2eb304
> RPC_READY:42:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1465206313
> heartbeat:44692819
> STATUS:16:NORMAL,-101785227522144798
> LOAD:44692800:3.18542273643E11
> SCHEMA:35090498:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:44692818:0.0
> NET_VERSION:1:9
> HOST_ID:2:0ec779f9-3ee4-41a2-80b2-f27f25521964
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1467201538
> heartbeat:38641336
> STATUS:16:NORMAL,-1004145257859511465
> LOAD:38641252:3.22115490439E11
> SCHEMA:29038976:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:38641338:0.0
> NET_VERSION:1:9
> HOST_ID:2:0ab413a9-5608-4d7b-97dc-6ed655e76dd0
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1467201918
> heartbeat:38640043
> STATUS:16:NORMAL,-1068518488491269501
> LOAD:38640012:3.30172166769E11
> SCHEMA:29037717:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:38640045:0.0
> NET_VERSION:1:9
> HOST_ID:2:7939c565-dc6e-43ad-b6ed-cfa3a19b20ab
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1479089944
> heartbeat:2582739
> STATUS:14:NORMAL,-109330739329126024
> LOAD:2582564:2.58856479776E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:2582738:0.0
> NET_VERSION:1:9
> HOST_ID:2:d2cf2695-25b7-4aad-92ab-9a0b593df5fb
> RPC_READY:42:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1474550583
> heartbeat:16351053
> STATUS:14:NORMAL,-1012946818076902491
> LOAD:16350923:3.32929676598E11
> SCHEMA:6748586:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:16351052:0.042265426367521286
> NET_VERSION:1:9
> HOST_ID:2:8e6417ec-e375-40a4-bb8d-3d99148846ff
> RPC_READY:42:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1464678970
> heartbeat:46292266
> STATUS:14:NORMAL,-100392380890457765
> LOAD:46292236:3.23397923906E11
> SCHEMA:36689950:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:46292265:0.0
> NET_VERSION:1:9
> HOST_ID:2:62323ea7-8d66-46fd-93e1-ff9bbf772c17
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1467202117
> heartbeat:38639452
> STATUS:16:NORMAL,-1023597752726656847
> LOAD:38639304:3.28121029924E11
> SCHEMA:29037141:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:38639451:0.0
> NET_VERSION:1:9
> HOST_ID:2:10b51458-800b-46f2-a80a-c2f2bbe1e46d
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1479087904
> heartbeat:2588919
> STATUS:14:NORMAL,-1013592683752719902
> LOAD:2588748:2.78416391781E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:2588918:0.0
> NET_VERSION:1:9
> HOST_ID:2:afcdc0a9-de5a-446d-868e-ee1dabfc0107
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1464513989
> heartbeat:46792633
> STATUS:14:NORMAL,-1063467021041506926
> LOAD:46792498:3.41276898418E11
> SCHEMA:37190355:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:46792632:0.0
> NET_VERSION:1:9
> HOST_ID:2:79a69fe9-de49-49d3-b8cb-fbc59e08a821
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1479088908
> heartbeat:2585894
> STATUS:14:NORMAL,-1060190437871147906
> LOAD:2585849:2.45716342224E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:2585893:0.0
> NET_VERSION:1:9
> HOST_ID:2:afc0ae8e-1126-426f-aa5b-0fce5578531c
> RPC_READY:44:true
> TOKENS:13:<hidden>
> ## result of nodetool gossipinfo on 1.1.1.2
> /x.x.x.x
> generation:1464511828
> heartbeat:46798699
> STATUS:14:NORMAL,-1022166408914492069
> LOAD:46798599:3.40042922062E11
> SCHEMA:37196731:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:46798698:0.0
> NET_VERSION:1:9
> HOST_ID:2:a08fbe58-779f-49f7-a0be-0a288d18c059
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /1.1.1.1
> generation:1471493280
> heartbeat:25624222
> STATUS:25498:NORMAL,-1023351769271276256
> LOAD:25624146:3.51052126949E11
> SCHEMA:16021810:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:1.1.1.1
> SEVERITY:25624221:0.0
> NET_VERSION:1:9
> HOST_ID:2:b79124e9-394c-4400-a8e7-a0c94aec6878
> RPC_READY:25524:true
> TOKENS:25497:<hidden>
> /x.x.x.x
> generation:1467205802
> heartbeat:38628440
> STATUS:14:NORMAL,-1124164164231176551
> LOAD:38628359:3.06404316564E11
> SCHEMA:29026044:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:38628439:0.0
> NET_VERSION:1:9
> HOST_ID:2:6dabba2e-085d-497b-a97e-b6ebf65258d6
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1479090215
> heartbeat:2581923
> STATUS:16:NORMAL,-1034930499164644146
> LOAD:2581842:2.60571265374E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:2581922:0.0
> NET_VERSION:1:9
> HOST_ID:2:a01d3304-5d8d-4ac9-935f-597ebf038e70
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1479089630
> heartbeat:2583690
> STATUS:16:NORMAL,-1104754836621297034
> LOAD:2583656:2.59767873976E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:2583689:0.0
> NET_VERSION:1:9
> HOST_ID:2:81d4608f-d1df-4f52-b104-5f1c7a3b4c96
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1471482865
> heartbeat:25655708
> STATUS:28484:NORMAL,-1005894856009564957
> LOAD:25655674:3.35493584628E11
> SCHEMA:16053376:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:25655707:0.0
> NET_VERSION:1:9
> HOST_ID:2:b39ee3e9-8f69-4f5c-8473-0e906c2301ca
> RPC_READY:28512:true
> TOKENS:28483:<hidden>
> /1.1.1.2
> generation:1479090411
> heartbeat:2581314
> STATUS:14:NORMAL,-1016248794247274800
> LOAD:2581277:2.71127001342E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:1.1.1.2
> SEVERITY:2581316:0.0
> NET_VERSION:1:9
> HOST_ID:2:8caa54f3-7d67-40d6-b224-8c64a1d289be
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1465163903
> heartbeat:44821587
> STATUS:14:NORMAL,-1077736070944113862
> LOAD:44821415:3.26343988237E11
> SCHEMA:35219220:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:44821586:0.0
> NET_VERSION:1:9
> HOST_ID:2:71db8e6a-cdbb-4924-860c-34e40731719f
> RPC_READY:42:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1467203191
> heartbeat:38636325
> STATUS:14:NORMAL,-100689147463310850
> LOAD:38636153:3.15710669743E11
> SCHEMA:29033962:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:38636327:0.0
> NET_VERSION:1:9
> HOST_ID:2:5582b905-7136-4e87-b1a9-d450ea2eb304
> RPC_READY:42:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1465206313
> heartbeat:44692816
> STATUS:16:NORMAL,-101785227522144798
> LOAD:44692800:3.18542273643E11
> SCHEMA:35090498:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:44692815:0.0
> NET_VERSION:1:9
> HOST_ID:2:0ec779f9-3ee4-41a2-80b2-f27f25521964
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1467201538
> heartbeat:38641336
> STATUS:16:NORMAL,-1004145257859511465
> LOAD:38641252:3.22115490439E11
> SCHEMA:29038976:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:38641338:0.0
> NET_VERSION:1:9
> HOST_ID:2:0ab413a9-5608-4d7b-97dc-6ed655e76dd0
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1467201918
> heartbeat:38640040
> STATUS:16:NORMAL,-1068518488491269501
> LOAD:38640012:3.30172166769E11
> SCHEMA:29037717:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:38640039:0.0
> NET_VERSION:1:9
> HOST_ID:2:7939c565-dc6e-43ad-b6ed-cfa3a19b20ab
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1479089944
> heartbeat:2582733
> STATUS:14:NORMAL,-109330739329126024
> LOAD:2582564:2.58856479776E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:2582732:0.042265426367521286
> NET_VERSION:1:9
> HOST_ID:2:d2cf2695-25b7-4aad-92ab-9a0b593df5fb
> RPC_READY:42:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1474550583
> heartbeat:16351050
> STATUS:14:NORMAL,-1012946818076902491
> LOAD:16350923:3.32929676598E11
> SCHEMA:6748586:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:16351049:0.0
> NET_VERSION:1:9
> HOST_ID:2:8e6417ec-e375-40a4-bb8d-3d99148846ff
> RPC_READY:42:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1464678970
> heartbeat:46292263
> STATUS:14:NORMAL,-100392380890457765
> LOAD:46292236:3.23397923906E11
> SCHEMA:36689950:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:46292265:0.0
> NET_VERSION:1:9
> HOST_ID:2:62323ea7-8d66-46fd-93e1-ff9bbf772c17
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1467202117
> heartbeat:38639449
> STATUS:16:NORMAL,-1023597752726656847
> LOAD:38639304:3.28121029924E11
> SCHEMA:29037141:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:38639451:0.0
> NET_VERSION:1:9
> HOST_ID:2:10b51458-800b-46f2-a80a-c2f2bbe1e46d
> RPC_READY:44:true
> TOKENS:15:<hidden>
> /x.x.x.x
> generation:1479087904
> heartbeat:2588919
> STATUS:14:NORMAL,-1013592683752719902
> LOAD:2588748:2.78416391781E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:2588918:0.0
> NET_VERSION:1:9
> HOST_ID:2:afcdc0a9-de5a-446d-868e-ee1dabfc0107
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1464513989
> heartbeat:46792633
> STATUS:14:NORMAL,-1063467021041506926
> LOAD:46792498:3.41276898418E11
> SCHEMA:37190355:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:46792632:0.0
> NET_VERSION:1:9
> HOST_ID:2:79a69fe9-de49-49d3-b8cb-fbc59e08a821
> RPC_READY:44:true
> TOKENS:13:<hidden>
> /x.x.x.x
> generation:1479088908
> heartbeat:2585891
> STATUS:14:NORMAL,-1060190437871147906
> LOAD:2585849:2.45716342224E11
> SCHEMA:10:c777bf0c-72ca-3bad-ac52-b9ad1ecd23b9
> DC:6:datacenter1
> RACK:8:rack1
> RELEASE_VERSION:4:2.2.5
> RPC_ADDRESS:3:x.x.x.x
> SEVERITY:2585890:0.0
> NET_VERSION:1:9
> HOST_ID:2:afc0ae8e-1126-426f-aa5b-0fce5578531c
> RPC_READY:44:true
> TOKENS:13:<hidden>
> {code}
> One thing we think of as weird is that 'messeage pending' and 'MUTATION drop'
> nodetool netstats
> {code}
> ## result of nodetool netstats on 1.1.1.1
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 0
> Pool Name Active Pending Completed
> Large messages n/a 7 2
> Small messages n/a 0 129707138163
> Gossip messages n/a 0 27177898
> ## result of nodetool netstats on 1.1.1.2
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 0
> Mismatch (Blocking): 0
> Mismatch (Background): 3
> Pool Name Active Pending Completed
> Large messages n/a 0 0
> Small messages n/a 484 19297343088
> Gossip messages n/a 0 2782060
> {code}
> nodetool tpstats
> {code}
> ## result of nodetool tpstats on 1.1.1.1
> Pool Name Active Pending Completed Blocked All time blocked
> Native-Transport-Requests 0 0 55687222731 0 4381472
> ReadRepairStage 0 0 70 0 0
> ReadStage 0 0 39737862 0 0
> RequestResponseStage 1 0 65682103003 0 0
> CounterMutationStage 0 0 0 0 0
> MutationStage 0 0 100679076772 0 0
> MemtablePostFlush 0 0 215235 0 0
> GossipStage 0 0 26510703 0 0
> MiscStage 0 0 0 0 0
> PendingRangeCalculator 0 0 37 0 0
> AntiEntropyStage 0 0 0 0 0
> CacheCleanupExecutor 0 0 184 0 0
> MigrationStage 0 0 59 0 0
> HintedHandoff 0 0 93 0 0
> ValidationExecutor 0 0 0 0 0
> MemtableFlushWriter 0 0 74239 0 0
> InternalResponseStage 0 0 3636382 0 0
> Sampler 0 0 0 0 0
> MemtableReclaimMemory 0 0 74239 0 0
> CompactionExecutor 0 0 5694753 0 0
> Message type Dropped
> RANGE_SLICE 0
> READ_REPAIR 0
> PAGED_RANGE 0
> READ 38268
> MUTATION 615972
> _TRACE 0
> REQUEST_RESPONSE 0
> COUNTER_MUTATION 0
> ## result of nodetool tpstats on 1.1.1.2
> Pool Name Active Pending Completed Blocked All time blocked
> Native-Transport-Requests 0 0 5783148174 0 970194
> ReadRepairStage 0 0 23 0 0
> ReadStage 0 0 227034 0 0
> RequestResponseStage 0 0 9656431051 0 0
> CounterMutationStage 0 0 0 0 0
> MutationStage 0 1 14494593737 0 0
> MemtablePostFlush 0 0 24720 0 0
> GossipStage 0 0 2667784 0 0
> MiscStage 0 0 0 0 0
> PendingRangeCalculator 0 0 26 0 0
> AntiEntropyStage 0 0 0 0 0
> CacheCleanupExecutor 0 0 0 0 0
> MigrationStage 0 0 0 0 0
> HintedHandoff 0 0 29 0 0
> ValidationExecutor 0 0 0 0 0
> MemtableFlushWriter 0 0 10555 0 0
> InternalResponseStage 0 0 3425759 0 0
> Sampler 0 0 0 0 0
> MemtableReclaimMemory 0 0 10555 0 0
> CompactionExecutor 2 3 551545 0 0
> Message type Dropped
> RANGE_SLICE 0
> READ_REPAIR 0
> PAGED_RANGE 0
> READ 0
> MUTATION 0
> _TRACE 0
> REQUEST_RESPONSE 0
> COUNTER_MUTATION 0
> {code}
> We found a similar issue that was reported. (https://issues.apache.org/jira/browse/CASSANDRA-4740)
> *But I think it caused by other reasons, due to problem was occurrence in recent months.*
> {code}
> ## on 1.1.1.1
> $ netstat -tn | grep 1.1.1.2
> tcp 0 0 1.1.1.1:50103 1.1.1.2:7000 ESTABLISHED
> tcp 0 0 1.1.1.1:7000 1.1.1.2:45946 ESTABLISHED
> tcp 0 0 1.1.1.1:38178 1.1.1.2:7000 ESTABLISHED
> tcp 0 0 1.1.1.1:7000 1.1.1.2:59958 ESTABLISHED
> ## on 1.1.1.2
> $ netstat -tn | grep 1.1.1.2
> tcp 0 0 1.1.1.2:7000 1.1.1.16:32973 ESTABLISHED
> tcp 0 0 1.1.1.2:34929 1.1.1.16:7000 ESTABLISHED
> tcp 0 50 1.1.1.2:59270 1.1.1.16:7000 ESTABLISHED
> tcp 0 0 1.1.1.2:7000 1.1.1.1:50103 ESTABLISHED
> tcp 0 0 1.1.1.2:45946 1.1.1.1:7000 ESTABLISHED
> tcp 0 0 1.1.1.2:59958 1.1.1.1:7000 ESTABLISHED
> tcp 0 0 1.1.1.2:7000 1.1.1.16:55680 ESTABLISHED
> tcp 0 0 1.1.1.2:7000 1.1.1.1:38178 ESTABLISHED
> {code}
> jvm and kernel are same on every nodes
> {code}
> $ uname -a
> Linux ip-10-211-195-229.ap-northeast-2.compute.internal 4.5.0-coreos-r1 #2 SMP Thu May 26 22:21:06 UTC 2016 x86_64 Linux
> $ java -version
> java version "1.8.0_102"
> Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
> $ which java
> /pang/program/jdk/bin/java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)