You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Joe Obernberger <jo...@gmail.com> on 2021/09/10 14:49:06 UTC
Unable to Gossip
I have a 10 node cluster and am trying to add another node. The new
node is running Rocky Linux and I'm getting the unable to gossip with
any peers error. Firewall and SELinux are off. I can ping all the
other nodes OK. I've checked everything I can think of (/etc/hosts,
listen_address, broadcast etc..). It all looks correct to me.
Any ideas? Could it be an incompatibility with Rocky?
DEBUG [main] 2021-09-10 06:45:24,846 YamlConfigurationLoader.java:112 -
Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
OutboundConnection.java:1150 -
/172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.253:7000-URGENT_MESSAGES-90efbb9e
successfully connected, version = 12, framing = LZ4, encryption =
unencrypted
INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
OutboundConnection.java:1150 -
/172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37:7000-URGENT_MESSAGES-eae47864
successfully connected, version = 12, framing = LZ4, encryption =
unencrypted
INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648 TokenMetadata.java:525
- Updating topology for all endpoints that have changed
DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
recording size estimates
ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
at
org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
StorageService.java:1621 - DRAINING: starting drain process
INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
HintsService.java:220 - Paused hints dispatch
WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
Gossiper.java:1993 - No local state, state is in silent shutdown, or
node hasn't joined, not announcing shutdown
Thank you!
-Joe
Re: Unable to Gossip
Posted by vytenis silgalis <vs...@gmail.com>.
Hmm. are the ports open on the `new` server?
Looks like it can connect to other nodes but other nodes can't connect to
it.
-Vy
On Fri, Sep 10, 2021 at 10:20 AM Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> Good idea.
> There are two seed nodes:
> I see this on one (note 172.16.100.44 is the new node):
>
> DEBUG [CompactionExecutor:1345] 2021-09-10 11:13:49,569
> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
> expired SSTables
> INFO [Messaging-EventLoop-3-10] 2021-09-10 11:14:22,810
> InboundConnectionInitiator.java:464 - /172.16.100.44:7000
> (/172.16.100.44:45970)->/172.16.100.253:7000-URGENT_MESSAGES-30a4fd82
> messaging connection established, version = 12, framing = LZ4, encryption =
> unencrypted
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 MessagingMetrics.java:206
> - GOSSIP_DIGEST_SYN messages were dropped in last 5000 ms: 0 internal and 1
> cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked
> All Time Blocked
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69 -
> ReadStage 0 0 4729810
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69 -
> CompactionExecutor 0 0 384171
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69 -
> MutationStage 0 0 14540487
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> MemtableReclaimMemory 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> PendingRangeCalculator 0 0 11
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> GossipStage 0 0 1126031
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> SecondaryIndexManagement 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> HintsDispatcher 0 0 15
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> Native-Transport-Requests 0 0 13286230
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> RequestResponseStage 0 0 15724485
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> MemtableFlushWriter 0 0 298
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> PerDiskMemtableFlushWriter_0 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
> MemtablePostFlush 0 0 336
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
> Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
> ValidationExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
> ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
> CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:79 -
> CompactionManager 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:91 -
> MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:101 -
> Cache Type Size Capacity
> KeysToSave
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:103 -
> KeyCache 75539240
> 104857600 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:109 -
> RowCache 0
> 0 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:116 -
> Table Memtable ops,data
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
> system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
> system_schema.types 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
> system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
> system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
> system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system_schema.views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system.compaction_history 2,403
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system.repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system.size_estimates 1625,31346
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
> system.table_estimates 3250,67852
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.paxos 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.built_views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.peer_events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.peers_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.peers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.peer_events_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.batches 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.transferred_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.transferred_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.view_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
> system.local 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system.sstable_activity 508,8564
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system.available_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system.prepared_statements 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_auth.network_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
> system_distributed.parent_repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> system_distributed.repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> system_distributed.view_build_status 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.node_operations 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.repair_run 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.repair_run_by_unit 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.diagnostic_event_subscription 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.running_reapers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.node_metrics_v3 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.repair_schedule_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
> reaper_db.repair_unit_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.repair_run_by_cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.schema_migration_leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.running_repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.schema_migration 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.node_metrics_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.repair_run_by_cluster_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> reaper_db.snapshot 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
> system_traces.sessions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> system_traces.events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.doc 71463,9507451
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.fieldcounts 719515,2487290
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.meta_ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.parent_child 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.blocks 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.ds_30 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.child_parent 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.seq 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.doctemp 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
> doc.ds_21 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.extractedmetadata 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.datasources 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ds_16 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ordered_fieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ds_22 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.correlationbuckets 39374,8377095
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.binary_docs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ordered_metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.ds_27 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
> doc.correlationdocbuckets 450045,39051043
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.ds_29 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.ds_32 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.ds_31 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.correlation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.ds_20 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.indexorganize 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.binary_data 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.origdoc 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.test_ds16_ds17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
> doc.metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.ds_28 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.sourcebuckets 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.ds_18 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.sourcecorrelationcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.ds_17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
> doc.sourcecorrelation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,579 MessagingMetrics.java:206
> - GOSSIP_DIGEST_SYN messages were dropped in last 5000 ms: 0 internal and 1
> cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked
> All Time Blocked
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> ReadStage 0 0 4729901
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> CompactionExecutor 0 0 384171
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> MutationStage 0 0 14540850
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> MemtableReclaimMemory 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> PendingRangeCalculator 0 0 11
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
> GossipStage 0 0 1126051
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> SecondaryIndexManagement 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> HintsDispatcher 0 0 15
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> Native-Transport-Requests 0 0 13286521
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> RequestResponseStage 0 0 15724799
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> MemtableFlushWriter 0 0 298
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> PerDiskMemtableFlushWriter_0 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> MemtablePostFlush 0 0 336
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
> ValidationExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69 -
> ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69 -
> CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:79 -
> CompactionManager 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:91 -
> MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:101 -
> Cache Type Size Capacity
> KeysToSave
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:103 -
> KeyCache 75540688
> 104857600 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:109 -
> RowCache 0
> 0 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:116 -
> Table Memtable ops,data
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
> system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
> system_schema.types 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
> system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
> system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system.compaction_history 2,403
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
> system.repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.size_estimates 1625,31346
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.table_estimates 3250,67852
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.paxos 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.built_views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.peer_events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.peers_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.peers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.peer_events_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
> system.batches 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.transferred_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.transferred_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.view_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.local 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.sstable_activity 508,8564
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.available_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system.prepared_statements 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
> system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_auth.network_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_distributed.parent_repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_distributed.repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> system_distributed.view_build_status 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> reaper_db.node_operations 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
> reaper_db.repair_run 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.repair_run_by_unit 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.diagnostic_event_subscription 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.running_reapers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.node_metrics_v3 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.repair_schedule_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.repair_unit_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.repair_run_by_cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
> reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.schema_migration_leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.running_repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.schema_migration 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.node_metrics_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.repair_run_by_cluster_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> reaper_db.snapshot 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> system_traces.sessions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
> system_traces.events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.doc 71523,9515801
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.fieldcounts 719639,2487714
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.meta_ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.parent_child 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.blocks 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.ds_30 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.child_parent 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.seq 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.doctemp 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
> doc.ds_21 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.extractedmetadata 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.datasources 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.ds_16 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.ordered_fieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.ds_22 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.correlationbuckets 39427,8388316
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.binary_docs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
> doc.ordered_metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.ds_27 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.correlationdocbuckets 450171,39061883
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.ds_29 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.ds_32 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.ds_31 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.correlation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.ds_20 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.indexorganize 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.binary_data 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
> doc.origdoc 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.test_ds16_ds17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.ds_28 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.sourcebuckets 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.ds_18 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.sourcecorrelationcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.ds_17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
> doc.sourcecorrelation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 MessagingMetrics.java:206
> - GOSSIP_DIGEST_SYN messages were dropped in last 5000 ms: 0 internal and 1
> cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:65 -
> Pool Name Active Pending Completed Blocked
> All Time Blocked
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> ReadStage 0 0 4730049
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> CompactionExecutor 0 0 384171
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> MutationStage 0 0 14541404
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> MemtableReclaimMemory 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> PendingRangeCalculator 0 0 11
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> GossipStage 0 0 1126070
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
> SecondaryIndexManagement 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> HintsDispatcher 0 0 15
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> Native-Transport-Requests 0 0 13286967
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> RequestResponseStage 0 0 15725284
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> MemtableFlushWriter 0 0 298
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> PerDiskMemtableFlushWriter_0 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> MemtablePostFlush 0 0 336
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> Sampler 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> ValidationExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> ViewBuildExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
> CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:79 -
> CompactionManager 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:91 -
> MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:101 -
> Cache Type Size Capacity
> KeysToSave
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:103 -
> KeyCache 75542896
> 104857600 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:109 -
> RowCache 0
> 0 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:116 -
> Table Memtable ops,data
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
> system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
> system_schema.types 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
> system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
> system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.compaction_history 2,403
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.size_estimates 1625,31346
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.table_estimates 3250,67852
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
> system.paxos 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.built_views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.peer_events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.peers_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.peers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.peer_events_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.batches 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.transferred_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.transferred_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.view_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.local 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.sstable_activity 508,8564
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.available_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
> system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system.prepared_statements 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_auth.network_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_distributed.parent_repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_distributed.repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> system_distributed.view_build_status 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
> reaper_db.node_operations 0,0
> .
> .
> .
>
> and this on the other:
>
> DEBUG [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,174
> InitialConnectionHandler.java:153 - Configured pipeline:
> DefaultChannelPipeline{(frameDecoder =
> org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
> org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
> org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
> org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
> WARN [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,180
> ExceptionHandlers.java:104 - Unknown exception in client networking
> io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
> Connection reset by peer
> DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,275
> InitialConnectionHandler.java:121 - Response to STARTUP sent, configuring
> pipeline for 5/v5
> DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,276
> InitialConnectionHandler.java:153 - Configured pipeline:
> DefaultChannelPipeline{(frameDecoder =
> org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
> org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
> org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
> org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
> WARN [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,278
> ExceptionHandlers.java:104 - Unknown exception in client networking
> io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
> Connection reset by peer
> INFO [Messaging-EventLoop-3-39] 2021-09-10 11:18:41,326
> InboundConnectionInitiator.java:464 - /172.16.100.44:7000
> (/172.16.100.44:44368)->/172.16.100.37:7000-URGENT_MESSAGES-8aa82849
> messaging connection established, version = 12, framing = LZ4, encryption =
> unencrypted
> INFO [ScheduledTasks:1] 2021-09-10 11:18:42,653 MessagingMetrics.java:206
> - GOSSIP_DIGEST_SYN messages were dropped in last 5000 ms: 0 internal and 1
> cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
> .
> .
> .
>
> -Joe
>
>
> On 9/10/2021 11:06 AM, Sam Tunnicliffe wrote:
>
> Is there anything in the logs of the other nodes, particularly those in
> the seeds list of the one which won't start? If this is a problem with the
> peers being unable to respond, as the issue fixed in 4.0.1
> (CASSANDRA-16877) was, you may see some indication there.
>
> Sam
>
> On 10 Sep 2021, at 15:56, Joe Obernberger <jo...@gmail.com>
> wrote:
>
> Thank you Jeff - yes, this is on the latest 4.0.1
>
> nodetool version
> ReleaseVersion: 4.0.1
> nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN 172.16.100.251 488.38 GiB 200 38.7%
> 660f476c-a124-4ca0-b55f-75efe56370da rack1
> UN 172.16.100.208 76.02 GiB 30 5.8%
> 2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
> UN 172.16.100.252 501.88 GiB 200 38.6%
> e83aa851-69b4-478f-88f6-60e657ea6539 rack1
> UN 172.16.100.249 517.27 GiB 200 38.6%
> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
> UN 172.16.100.36 524.45 GiB 200 38.6%
> d9702f96-256e-45ae-8e12-69a42712be50 rack1
> UN 172.16.100.39 521.05 GiB 200 38.6%
> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
> UN 172.16.100.253 11.39 GiB 4 0.8%
> a1a16910-9167-4174-b34b-eb859d36347e rack1
> UN 172.16.100.248 524.46 GiB 200 38.7%
> 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
> UN 172.16.100.37 314.67 GiB 120 23.2%
> 08a19658-40be-4e55-8709-812b3d4ac750 rack1
> UN 172.16.100.250 464.23 GiB 200 38.6%
> b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>
> yum list installed | grep cass
> cassandra.noarch 4.0.1-1
> @cassandra
>
> -Joe
> On 9/10/2021 10:54 AM, Jeff Jirsa wrote:
>
> Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is too
> large for the urgent message queue, causing this stack trace, and was
> released 3 days ago. I've never seen it on a 10 node cluster before, but
> I'd be trying that.
>
> On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger <
> joseph.obernberger@gmail.com> wrote:
>
>> I have a 10 node cluster and am trying to add another node. The new
>> node is running Rocky Linux and I'm getting the unable to gossip with
>> any peers error. Firewall and SELinux are off. I can ping all the
>> other nodes OK. I've checked everything I can think of (/etc/hosts,
>> listen_address, broadcast etc..). It all looks correct to me.
>> Any ideas? Could it be an incompatibility with Rocky?
>>
>> DEBUG [main] 2021-09-10 06:45:24,846 YamlConfigurationLoader.java:112 -
>> Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
>> INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
>> OutboundConnection.java:1150 -
>> /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.
>> <http://172.16.0.100/>253:7000-URGENT_MESSAGES-90efbb9e
>> successfully connected, version = 12, framing = LZ4, encryption =
>> unencrypted
>> INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
>> OutboundConnection.java:1150 -
>> /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37:7000-URGENT_MESSAGES-eae47864
>>
>> successfully connected, version = 12, framing = LZ4, encryption =
>> unencrypted
>> INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648 TokenMetadata.java:525
>> - Updating topology for all endpoints that have changed
>> DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
>> SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
>> recording size estimates
>> ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
>> Exception encountered during startup
>> java.lang.RuntimeException: Unable to gossip with any peers
>> at
>> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
>> at
>>
>> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
>> at
>>
>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
>> at
>>
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
>> at
>>
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
>> at
>>
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
>> at
>>
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
>> at
>>
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
>> DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
>> StorageService.java:1621 - DRAINING: starting drain process
>> INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
>> HintsService.java:220 - Paused hints dispatch
>> WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
>> Gossiper.java:1993 - No local state, state is in silent shutdown, or
>> node hasn't joined, not announcing shutdown
>>
>> Thank you!
>>
>> -Joe
>>
>>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
>
Re: Unable to Gossip
Posted by Joe Obernberger <jo...@gmail.com>.
Oh! Excellent! Doh! That was it. So when we add a new system, we use
puppet to push things out...like NTP...well this is our first Rocky
Linux install and guess what I didn't do?
Thank you Song. The new machine is now joining the cluster.
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 172.16.100.251 490.67 GiB 200 38.7%
660f476c-a124-4ca0-b55f-75efe56370da rack1
UN 172.16.100.208 76.31 GiB 30 5.8%
2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
UN 172.16.100.252 504.13 GiB 200 38.6%
e83aa851-69b4-478f-88f6-60e657ea6539 rack1
UN 172.16.100.249 519.29 GiB 200 38.6%
49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
UN 172.16.100.36 526.47 GiB 200 38.6%
d9702f96-256e-45ae-8e12-69a42712be50 rack1
UN 172.16.100.39 523.19 GiB 200 38.6%
93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
UN 172.16.100.253 11.42 GiB 4 0.8%
a1a16910-9167-4174-b34b-eb859d36347e rack1
UN 172.16.100.248 526.61 GiB 200 38.7%
4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
*UJ 172.16.100.44 179.98 KiB 200 ?
b2e5366e-8386-40ec-a641-27944a5a7cfa rack1*
UN 172.16.100.37 315.89 GiB 120 23.2%
08a19658-40be-4e55-8709-812b3d4ac750 rack1
UN 172.16.100.250 465.48 GiB 200 38.6%
b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
Cheers!
-Joe
On 9/10/2021 1:25 PM, Bowen Song wrote:
>
> Hello Joe,
>
>
> These logs indicate the clocks are out of sync (by over 4.2 hours)
> between the new node and the seed nodes:
>
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567
> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were
> dropped in last 5000 ms: 0 internal and 1 cross node. Mean
> internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594
> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were
> dropped in last 5000 ms: 0 internal and 1 cross node. Mean
> internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:18:42,653
> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were
> dropped in last 5000 ms: 0 internal and 1 cross node. Mean
> internal dropped latency: 0 ms and Mean cross-node dropped
> latency: 15137813 ms
>
> Can you please check that the NTP client is running on all servers and
> the clocks are in sync?
>
>
> Cheers,
>
> Bowen
>
>
>
> On 10/09/2021 16:18, Joe Obernberger wrote:
>>
>> Good idea.
>> There are two seed nodes:
>> I see this on one (note 172.16.100.44 is the new node):
>>
>> DEBUG [CompactionExecutor:1345] 2021-09-10 11:13:49,569
>> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
>> expired SSTables
>> INFO [Messaging-EventLoop-3-10] 2021-09-10 11:14:22,810
>> InboundConnectionInitiator.java:464 -
>> /172.16.100.44:7000(/172.16.100.44:45970)->/172.16.100.253:7000-URGENT_MESSAGES-30a4fd82
>> messaging connection established, version = 12, framing = LZ4,
>> encryption = unencrypted
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567
>> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
>> in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
>> latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:65
>> - Pool Name Active Pending Completed Blocked All Time Blocked
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69
>> - ReadStage 0 0 4729810 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69
>> - CompactionExecutor 0 0 384171
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69
>> - MutationStage 0 0 14540487 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
>> - MemtableReclaimMemory 0 0 316
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
>> - PendingRangeCalculator 0 0 11
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
>> - GossipStage 0 0 1126031 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
>> - SecondaryIndexManagement 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
>> - HintsDispatcher 0 0 15 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
>> - Native-Transport-Requests 0 0 13286230
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
>> - RequestResponseStage 0 0 15724485
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
>> - MemtableFlushWriter 0 0 298
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
>> - PerDiskMemtableFlushWriter_0 0 0 316
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
>> - MemtablePostFlush 0 0 336
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
>> - Sampler 0 0 0 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
>> - ValidationExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
>> - ViewBuildExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
>> - CacheCleanupExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:79
>> - CompactionManager 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:91
>> - MessagingService n/a 0/0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570
>> StatusLogger.java:101 - Cache Type Size
>> Capacity KeysToSave
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570
>> StatusLogger.java:103 - KeyCache 75539240
>> 104857600 all
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570
>> StatusLogger.java:109 - RowCache 0
>> 0 all
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570
>> StatusLogger.java:116 - Table Memtable ops,data
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570
>> StatusLogger.java:119 - system_schema.columns 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570
>> StatusLogger.java:119 - system_schema.types 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570
>> StatusLogger.java:119 - system_schema.indexes 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570
>> StatusLogger.java:119 - system_schema.keyspaces 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570
>> StatusLogger.java:119 - system_schema.dropped_columns 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571
>> StatusLogger.java:119 - system_schema.aggregates 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571
>> StatusLogger.java:119 - system_schema.triggers 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571
>> StatusLogger.java:119 - system_schema.tables 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571
>> StatusLogger.java:119 - system_schema.views 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571
>> StatusLogger.java:119 - system_schema.functions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571
>> StatusLogger.java:119 - system.compaction_history 2,403
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571
>> StatusLogger.java:119 - system.IndexInfo 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571
>> StatusLogger.java:119 - system.repairs 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571
>> StatusLogger.java:119 - system.size_estimates 1625,31346
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571
>> StatusLogger.java:119 - system.table_estimates 3250,67852
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 - system.paxos 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 - system.built_views 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 - system.peer_events 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 - system.peers_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 - system.peers 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 - system.peer_events_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 - system.batches 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 - system.transferred_ranges 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 - system.transferred_ranges_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 -
>> system.view_builds_in_progress 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572
>> StatusLogger.java:119 - system.local 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573
>> StatusLogger.java:119 - system.sstable_activity 508,8564
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573
>> StatusLogger.java:119 - system.available_ranges_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573
>> StatusLogger.java:119 - system.available_ranges 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573
>> StatusLogger.java:119 - system.prepared_statements 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573
>> StatusLogger.java:119 - system_auth.roles 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573
>> StatusLogger.java:119 - system_auth.role_members 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573
>> StatusLogger.java:119 -
>> system_auth.resource_role_permissons_index 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573
>> StatusLogger.java:119 -
>> system_auth.network_permissions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573
>> StatusLogger.java:119 - system_auth.role_permissions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573
>> StatusLogger.java:119 -
>> system_distributed.parent_repair_history 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 -
>> system_distributed.repair_history 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 -
>> system_distributed.view_build_status 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 - reaper_db.node_operations 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 - reaper_db.repair_run 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 - reaper_db.repair_run_by_unit 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 - reaper_db.cluster 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 -
>> reaper_db.diagnostic_event_subscription 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 - reaper_db.running_reapers 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 - reaper_db.node_metrics_v3 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 - reaper_db.repair_schedule_v1 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574
>> StatusLogger.java:119 - reaper_db.repair_unit_v1 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575
>> StatusLogger.java:119 -
>> reaper_db.repair_run_by_cluster 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575
>> StatusLogger.java:119 -
>> reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575
>> StatusLogger.java:119 - reaper_db.leader 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575
>> StatusLogger.java:119 -
>> reaper_db.schema_migration_leader 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575
>> StatusLogger.java:119 - reaper_db.running_repairs 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575
>> StatusLogger.java:119 - reaper_db.schema_migration 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575
>> StatusLogger.java:119 - reaper_db.node_metrics_v1 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575
>> StatusLogger.java:119 -
>> reaper_db.repair_run_by_cluster_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575
>> StatusLogger.java:119 - reaper_db.snapshot 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575
>> StatusLogger.java:119 - system_traces.sessions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - system_traces.events 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - doc.doc 71463,9507451
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - doc.fieldcounts 719515,2487290
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - doc.meta_ftametrics 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - doc.parent_child 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - doc.blocks 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - doc.ds_30 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - doc.child_parent 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - doc.seq 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - doc.doctemp 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576
>> StatusLogger.java:119 - doc.ds_21 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.extractedmetadata 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.datasources 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.ds_16 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.ordered_fieldcounts 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.ds_22 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.correlationbuckets 39374,8377095
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.ftametrics 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.binary_docs 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.ordered_metafieldcounts 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.ds_27 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577
>> StatusLogger.java:119 - doc.correlationdocbuckets 450045,39051043
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578
>> StatusLogger.java:119 - doc.ds_29 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578
>> StatusLogger.java:119 - doc.ds_32 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578
>> StatusLogger.java:119 - doc.ds_31 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578
>> StatusLogger.java:119 - doc.correlation 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578
>> StatusLogger.java:119 - doc.ds_20 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578
>> StatusLogger.java:119 - doc.indexorganize 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578
>> StatusLogger.java:119 - doc.binary_data 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578
>> StatusLogger.java:119 - doc.origdoc 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578
>> StatusLogger.java:119 - doc.test_ds16_ds17 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578
>> StatusLogger.java:119 - doc.metafieldcounts 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579
>> StatusLogger.java:119 - doc.ds_28 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579
>> StatusLogger.java:119 - doc.sourcebuckets 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579
>> StatusLogger.java:119 - doc.ds_18 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579
>> StatusLogger.java:119 - doc.sourcecorrelationcounts 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579
>> StatusLogger.java:119 - doc.ds_17 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579
>> StatusLogger.java:119 - doc.sourcecorrelation 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,579
>> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
>> in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
>> latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:65
>> - Pool Name Active Pending Completed Blocked All Time Blocked
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
>> - ReadStage 0 0 4729901 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
>> - CompactionExecutor 0 0 384171
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
>> - MutationStage 0 0 14540850 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
>> - MemtableReclaimMemory 0 0 316
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
>> - PendingRangeCalculator 0 0 11
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
>> - GossipStage 0 0 1126051 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
>> - SecondaryIndexManagement 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
>> - HintsDispatcher 0 0 15 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
>> - Native-Transport-Requests 0 0 13286521
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
>> - RequestResponseStage 0 0 15724799
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
>> - MemtableFlushWriter 0 0 298
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
>> - PerDiskMemtableFlushWriter_0 0 0 316
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
>> - MemtablePostFlush 0 0 336
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
>> - Sampler 0 0 0 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
>> - ValidationExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69
>> - ViewBuildExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69
>> - CacheCleanupExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:79
>> - CompactionManager 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:91
>> - MessagingService n/a 0/0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583
>> StatusLogger.java:101 - Cache Type Size
>> Capacity KeysToSave
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583
>> StatusLogger.java:103 - KeyCache 75540688
>> 104857600 all
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583
>> StatusLogger.java:109 - RowCache 0
>> 0 all
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583
>> StatusLogger.java:116 - Table Memtable ops,data
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583
>> StatusLogger.java:119 - system_schema.columns 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583
>> StatusLogger.java:119 - system_schema.types 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583
>> StatusLogger.java:119 - system_schema.indexes 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583
>> StatusLogger.java:119 - system_schema.keyspaces 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584
>> StatusLogger.java:119 - system_schema.dropped_columns 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584
>> StatusLogger.java:119 - system_schema.aggregates 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584
>> StatusLogger.java:119 - system_schema.triggers 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584
>> StatusLogger.java:119 - system_schema.tables 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584
>> StatusLogger.java:119 - system_schema.views 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584
>> StatusLogger.java:119 - system_schema.functions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584
>> StatusLogger.java:119 - system.compaction_history 2,403
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584
>> StatusLogger.java:119 - system.IndexInfo 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584
>> StatusLogger.java:119 - system.repairs 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585
>> StatusLogger.java:119 - system.size_estimates 1625,31346
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585
>> StatusLogger.java:119 - system.table_estimates 3250,67852
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585
>> StatusLogger.java:119 - system.paxos 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585
>> StatusLogger.java:119 - system.built_views 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585
>> StatusLogger.java:119 - system.peer_events 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585
>> StatusLogger.java:119 - system.peers_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585
>> StatusLogger.java:119 - system.peers 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585
>> StatusLogger.java:119 - system.peer_events_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585
>> StatusLogger.java:119 - system.batches 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586
>> StatusLogger.java:119 - system.transferred_ranges 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586
>> StatusLogger.java:119 - system.transferred_ranges_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586
>> StatusLogger.java:119 -
>> system.view_builds_in_progress 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586
>> StatusLogger.java:119 - system.local 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586
>> StatusLogger.java:119 - system.sstable_activity 508,8564
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586
>> StatusLogger.java:119 - system.available_ranges_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586
>> StatusLogger.java:119 - system.available_ranges 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586
>> StatusLogger.java:119 - system.prepared_statements 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586
>> StatusLogger.java:119 - system_auth.roles 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587
>> StatusLogger.java:119 - system_auth.role_members 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587
>> StatusLogger.java:119 -
>> system_auth.resource_role_permissons_index 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587
>> StatusLogger.java:119 -
>> system_auth.network_permissions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587
>> StatusLogger.java:119 - system_auth.role_permissions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587
>> StatusLogger.java:119 -
>> system_distributed.parent_repair_history 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587
>> StatusLogger.java:119 -
>> system_distributed.repair_history 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587
>> StatusLogger.java:119 -
>> system_distributed.view_build_status 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587
>> StatusLogger.java:119 - reaper_db.node_operations 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587
>> StatusLogger.java:119 - reaper_db.repair_run 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588
>> StatusLogger.java:119 - reaper_db.repair_run_by_unit 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588
>> StatusLogger.java:119 - reaper_db.cluster 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588
>> StatusLogger.java:119 -
>> reaper_db.diagnostic_event_subscription 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588
>> StatusLogger.java:119 - reaper_db.running_reapers 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588
>> StatusLogger.java:119 - reaper_db.node_metrics_v3 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588
>> StatusLogger.java:119 - reaper_db.repair_schedule_v1 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588
>> StatusLogger.java:119 - reaper_db.repair_unit_v1 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588
>> StatusLogger.java:119 -
>> reaper_db.repair_run_by_cluster 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588
>> StatusLogger.java:119 -
>> reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589
>> StatusLogger.java:119 - reaper_db.leader 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589
>> StatusLogger.java:119 -
>> reaper_db.schema_migration_leader 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589
>> StatusLogger.java:119 - reaper_db.running_repairs 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589
>> StatusLogger.java:119 - reaper_db.schema_migration 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589
>> StatusLogger.java:119 - reaper_db.node_metrics_v1 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589
>> StatusLogger.java:119 -
>> reaper_db.repair_run_by_cluster_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589
>> StatusLogger.java:119 - reaper_db.snapshot 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589
>> StatusLogger.java:119 - system_traces.sessions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589
>> StatusLogger.java:119 - system_traces.events 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590
>> StatusLogger.java:119 - doc.doc 71523,9515801
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590
>> StatusLogger.java:119 - doc.fieldcounts 719639,2487714
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590
>> StatusLogger.java:119 - doc.meta_ftametrics 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590
>> StatusLogger.java:119 - doc.parent_child 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590
>> StatusLogger.java:119 - doc.blocks 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590
>> StatusLogger.java:119 - doc.ds_30 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590
>> StatusLogger.java:119 - doc.child_parent 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590
>> StatusLogger.java:119 - doc.seq 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590
>> StatusLogger.java:119 - doc.doctemp 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590
>> StatusLogger.java:119 - doc.ds_21 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591
>> StatusLogger.java:119 - doc.extractedmetadata 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591
>> StatusLogger.java:119 - doc.datasources 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591
>> StatusLogger.java:119 - doc.ds_16 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591
>> StatusLogger.java:119 - doc.ordered_fieldcounts 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591
>> StatusLogger.java:119 - doc.ds_22 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591
>> StatusLogger.java:119 - doc.correlationbuckets 39427,8388316
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591
>> StatusLogger.java:119 - doc.ftametrics 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591
>> StatusLogger.java:119 - doc.binary_docs 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591
>> StatusLogger.java:119 - doc.ordered_metafieldcounts 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592
>> StatusLogger.java:119 - doc.ds_27 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592
>> StatusLogger.java:119 - doc.correlationdocbuckets 450171,39061883
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592
>> StatusLogger.java:119 - doc.ds_29 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592
>> StatusLogger.java:119 - doc.ds_32 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592
>> StatusLogger.java:119 - doc.ds_31 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592
>> StatusLogger.java:119 - doc.correlation 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592
>> StatusLogger.java:119 - doc.ds_20 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592
>> StatusLogger.java:119 - doc.indexorganize 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592
>> StatusLogger.java:119 - doc.binary_data 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592
>> StatusLogger.java:119 - doc.origdoc 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593
>> StatusLogger.java:119 - doc.test_ds16_ds17 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593
>> StatusLogger.java:119 - doc.metafieldcounts 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593
>> StatusLogger.java:119 - doc.ds_28 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593
>> StatusLogger.java:119 - doc.sourcebuckets 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593
>> StatusLogger.java:119 - doc.ds_18 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593
>> StatusLogger.java:119 - doc.sourcecorrelationcounts 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593
>> StatusLogger.java:119 - doc.ds_17 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593
>> StatusLogger.java:119 - doc.sourcecorrelation 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594
>> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
>> in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
>> latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:65
>> - Pool Name Active Pending Completed Blocked All Time Blocked
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
>> - ReadStage 0 0 4730049 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
>> - CompactionExecutor 0 0 384171
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
>> - MutationStage 0 0 14541404 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
>> - MemtableReclaimMemory 0 0 316
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
>> - PendingRangeCalculator 0 0 11
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
>> - GossipStage 0 0 1126070 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
>> - SecondaryIndexManagement 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
>> - HintsDispatcher 0 0 15 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
>> - Native-Transport-Requests 0 0 13286967
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
>> - RequestResponseStage 0 0 15725284
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
>> - MemtableFlushWriter 0 0 298
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
>> - PerDiskMemtableFlushWriter_0 0 0 316
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
>> - MemtablePostFlush 0 0 336
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
>> - Sampler 0 0 0 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
>> - ValidationExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
>> - ViewBuildExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
>> - CacheCleanupExecutor 0 0 0
>> 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:79
>> - CompactionManager 0 0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:91
>> - MessagingService n/a 0/0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596
>> StatusLogger.java:101 - Cache Type Size
>> Capacity KeysToSave
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596
>> StatusLogger.java:103 - KeyCache 75542896
>> 104857600 all
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596
>> StatusLogger.java:109 - RowCache 0
>> 0 all
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596
>> StatusLogger.java:116 - Table Memtable ops,data
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596
>> StatusLogger.java:119 - system_schema.columns 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596
>> StatusLogger.java:119 - system_schema.types 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596
>> StatusLogger.java:119 - system_schema.indexes 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596
>> StatusLogger.java:119 - system_schema.keyspaces 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system_schema.dropped_columns 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system_schema.aggregates 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system_schema.triggers 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system_schema.tables 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system_schema.views 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system_schema.functions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system.compaction_history 2,403
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system.IndexInfo 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system.repairs 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system.size_estimates 1625,31346
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system.table_estimates 3250,67852
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597
>> StatusLogger.java:119 - system.paxos 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.built_views 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.peer_events 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.peers_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.peers 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.peer_events_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.batches 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.transferred_ranges 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.transferred_ranges_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 -
>> system.view_builds_in_progress 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.local 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.sstable_activity 508,8564
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.available_ranges_v2 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598
>> StatusLogger.java:119 - system.available_ranges 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599
>> StatusLogger.java:119 - system.prepared_statements 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599
>> StatusLogger.java:119 - system_auth.roles 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599
>> StatusLogger.java:119 - system_auth.role_members 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599
>> StatusLogger.java:119 -
>> system_auth.resource_role_permissons_index 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599
>> StatusLogger.java:119 -
>> system_auth.network_permissions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599
>> StatusLogger.java:119 - system_auth.role_permissions 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599
>> StatusLogger.java:119 -
>> system_distributed.parent_repair_history 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599
>> StatusLogger.java:119 -
>> system_distributed.repair_history 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599
>> StatusLogger.java:119 -
>> system_distributed.view_build_status 0,0
>> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599
>> StatusLogger.java:119 - reaper_db.node_operations 0,0
>> .
>> .
>> .
>>
>> and this on the other:
>>
>> DEBUG [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,174
>> InitialConnectionHandler.java:153 - Configured pipeline:
>> DefaultChannelPipeline{(frameDecoder =
>> org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
>> org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
>> org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler
>> =
>> org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
>> WARN [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,180
>> ExceptionHandlers.java:104 - Unknown exception in client networking
>> io.netty.channel.unix.Errors$NativeIoException: readAddress(..)
>> failed: Connection reset by peer
>> DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,275
>> InitialConnectionHandler.java:121 - Response to STARTUP sent,
>> configuring pipeline for 5/v5
>> DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,276
>> InitialConnectionHandler.java:153 - Configured pipeline:
>> DefaultChannelPipeline{(frameDecoder =
>> org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
>> org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
>> org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler
>> =
>> org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
>> WARN [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,278
>> ExceptionHandlers.java:104 - Unknown exception in client networking
>> io.netty.channel.unix.Errors$NativeIoException: readAddress(..)
>> failed: Connection reset by peer
>> INFO [Messaging-EventLoop-3-39] 2021-09-10 11:18:41,326
>> InboundConnectionInitiator.java:464 -
>> /172.16.100.44:7000(/172.16.100.44:44368)->/172.16.100.37:7000-URGENT_MESSAGES-8aa82849
>> messaging connection established, version = 12, framing = LZ4,
>> encryption = unencrypted
>> INFO [ScheduledTasks:1] 2021-09-10 11:18:42,653
>> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
>> in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
>> latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
>> .
>> .
>> .
>>
>> -Joe
>>
>>
>> On 9/10/2021 11:06 AM, Sam Tunnicliffe wrote:
>>> Is there anything in the logs of the other nodes, particularly those
>>> in the seeds list of the one which won't start? If this is a problem
>>> with the peers being unable to respond, as the issue fixed in 4.0.1
>>> (CASSANDRA-16877) was, you may see some indication there.
>>>
>>> Sam
>>>
>>>> On 10 Sep 2021, at 15:56, Joe Obernberger
>>>> <jo...@gmail.com> wrote:
>>>>
>>>> Thank you Jeff - yes, this is on the latest 4.0.1
>>>>
>>>> nodetool version
>>>> ReleaseVersion: 4.0.1
>>>> nodetool status
>>>> Datacenter: datacenter1
>>>> =======================
>>>> Status=Up/Down
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>> -- Address Load Tokens Owns (effective) Host
>>>> ID Rack
>>>> UN 172.16.100.251 488.38 GiB 200 38.7%
>>>> 660f476c-a124-4ca0-b55f-75efe56370da rack1
>>>> UN 172.16.100.208 76.02 GiB 30 5.8%
>>>> 2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
>>>> UN 172.16.100.252 501.88 GiB 200 38.6%
>>>> e83aa851-69b4-478f-88f6-60e657ea6539 rack1
>>>> UN 172.16.100.249 517.27 GiB 200 38.6%
>>>> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
>>>> UN 172.16.100.36 524.45 GiB 200 38.6%
>>>> d9702f96-256e-45ae-8e12-69a42712be50 rack1
>>>> UN 172.16.100.39 521.05 GiB 200 38.6%
>>>> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
>>>> UN 172.16.100.253 11.39 GiB 4 0.8%
>>>> a1a16910-9167-4174-b34b-eb859d36347e rack1
>>>> UN 172.16.100.248 524.46 GiB 200 38.7%
>>>> 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
>>>> UN 172.16.100.37 314.67 GiB 120 23.2%
>>>> 08a19658-40be-4e55-8709-812b3d4ac750 rack1
>>>> UN 172.16.100.250 464.23 GiB 200 38.6%
>>>> b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>>>>
>>>> yum list installed | grep cass
>>>> cassandra.noarch 4.0.1-1 @cassandra
>>>>
>>>> -Joe
>>>>
>>>> On 9/10/2021 10:54 AM, Jeff Jirsa wrote:
>>>>> Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is
>>>>> too large for the urgent message queue, causing this stack trace,
>>>>> and was released 3 days ago. I've never seen it on a 10 node
>>>>> cluster before, but I'd be trying that.
>>>>>
>>>>> On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger
>>>>> <jo...@gmail.com> wrote:
>>>>>
>>>>> I have a 10 node cluster and am trying to add another node.
>>>>> The new
>>>>> node is running Rocky Linux and I'm getting the unable to
>>>>> gossip with
>>>>> any peers error. Firewall and SELinux are off. I can ping
>>>>> all the
>>>>> other nodes OK. I've checked everything I can think of
>>>>> (/etc/hosts,
>>>>> listen_address, broadcast etc..). It all looks correct to me.
>>>>> Any ideas? Could it be an incompatibility with Rocky?
>>>>>
>>>>> DEBUG [main] 2021-09-10 06:45:24,846
>>>>> YamlConfigurationLoader.java:112 -
>>>>> Loading settings from
>>>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>>>> INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
>>>>> OutboundConnection.java:1150 -
>>>>> /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.
>>>>> <http://172.16.0.100/>253:7000-URGENT_MESSAGES-90efbb9e
>>>>> successfully connected, version = 12, framing = LZ4, encryption =
>>>>> unencrypted
>>>>> INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
>>>>> OutboundConnection.java:1150 -
>>>>> /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37
>>>>> <http://172.16.100.37/>:7000-URGENT_MESSAGES-eae47864
>>>>> successfully connected, version = 12, framing = LZ4, encryption =
>>>>> unencrypted
>>>>> INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648
>>>>> TokenMetadata.java:525
>>>>> - Updating topology for all endpoints that have changed
>>>>> DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
>>>>> SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
>>>>> recording size estimates
>>>>> ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
>>>>> Exception encountered during startup
>>>>> java.lang.RuntimeException: Unable to gossip with any peers
>>>>> at
>>>>> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
>>>>> DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
>>>>> StorageService.java:1621 - DRAINING: starting drain process
>>>>> INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
>>>>> HintsService.java:220 - Paused hints dispatch
>>>>> WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
>>>>> Gossiper.java:1993 - No local state, state is in silent
>>>>> shutdown, or
>>>>> node hasn't joined, not announcing shutdown
>>>>>
>>>>> Thank you!
>>>>>
>>>>> -Joe
>>>>>
>>>>>
>>>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>>>> Virus-free. www.avg.com
>>>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>>>>
>>>>>
>>>>> <x-msg://5/#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
Re: Unable to Gossip
Posted by Bowen Song <bo...@bso.ng>.
Hello Joe,
These logs indicate the clocks are out of sync (by over 4.2 hours)
between the new node and the seed nodes:
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
INFO [ScheduledTasks:1] 2021-09-10 11:18:42,653
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped
in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
Can you please check that the NTP client is running on all servers and
the clocks are in sync?
Cheers,
Bowen
On 10/09/2021 16:18, Joe Obernberger wrote:
>
> Good idea.
> There are two seed nodes:
> I see this on one (note 172.16.100.44 is the new node):
>
> DEBUG [CompactionExecutor:1345] 2021-09-10 11:13:49,569
> TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
> expired SSTables
> INFO [Messaging-EventLoop-3-10] 2021-09-10 11:14:22,810
> InboundConnectionInitiator.java:464 -
> /172.16.100.44:7000(/172.16.100.44:45970)->/172.16.100.253:7000-URGENT_MESSAGES-30a4fd82
> messaging connection established, version = 12, framing = LZ4,
> encryption = unencrypted
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567
> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
> last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
> latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:65
> - Pool Name Active Pending Completed
> Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69
> - ReadStage 0 0 4729810 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69
> - CompactionExecutor 0 0 384171
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69
> - MutationStage 0 0 14540487 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
> - MemtableReclaimMemory 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
> - PendingRangeCalculator 0 0 11
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
> - GossipStage 0 0 1126031 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
> - SecondaryIndexManagement 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
> - HintsDispatcher 0 0 15 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
> - Native-Transport-Requests 0 0 13286230
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
> - RequestResponseStage 0 0 15724485
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
> - MemtableFlushWriter 0 0 298
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
> - PerDiskMemtableFlushWriter_0 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69
> - MemtablePostFlush 0 0 336 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
> - Sampler 0 0 0 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
> - ValidationExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
> - ViewBuildExecutor 0 0 0 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69
> - CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:79
> - CompactionManager 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:91
> - MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:101
> - Cache Type Size Capacity KeysToSave
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:103
> - KeyCache 75539240 104857600 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:109
> - RowCache 0 0 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:116
> - Table Memtable ops,data
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119
> - system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119
> - system_schema.types 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119
> - system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119
> - system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119
> - system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
> - system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
> - system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
> - system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
> - system_schema.views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
> - system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
> - system.compaction_history 2,403
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
> - system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
> - system.repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
> - system.size_estimates 1625,31346
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119
> - system.table_estimates 3250,67852
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.paxos 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.built_views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.peer_events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.peers_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.peers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.peer_events_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.batches 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.transferred_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.transferred_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.view_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119
> - system.local 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
> - system.sstable_activity 508,8564
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
> - system.available_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
> - system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
> - system.prepared_statements 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
> - system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
> - system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
> - system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
> - system_auth.network_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
> - system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119
> - system_distributed.parent_repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - system_distributed.repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - system_distributed.view_build_status 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - reaper_db.node_operations 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - reaper_db.repair_run 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - reaper_db.repair_run_by_unit 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - reaper_db.cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - reaper_db.diagnostic_event_subscription 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - reaper_db.running_reapers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - reaper_db.node_metrics_v3 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - reaper_db.repair_schedule_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119
> - reaper_db.repair_unit_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
> - reaper_db.repair_run_by_cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
> - reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
> - reaper_db.leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
> - reaper_db.schema_migration_leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
> - reaper_db.running_repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
> - reaper_db.schema_migration 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
> - reaper_db.node_metrics_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
> - reaper_db.repair_run_by_cluster_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
> - reaper_db.snapshot 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119
> - system_traces.sessions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - system_traces.events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - doc.doc 71463,9507451
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - doc.fieldcounts 719515,2487290
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - doc.meta_ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - doc.parent_child 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - doc.blocks 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - doc.ds_30 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - doc.child_parent 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - doc.seq 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - doc.doctemp 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119
> - doc.ds_21 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.extractedmetadata 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.datasources 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.ds_16 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.ordered_fieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.ds_22 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.correlationbuckets 39374,8377095
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.binary_docs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.ordered_metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.ds_27 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119
> - doc.correlationdocbuckets 450045,39051043
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
> - doc.ds_29 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
> - doc.ds_32 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
> - doc.ds_31 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
> - doc.correlation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
> - doc.ds_20 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
> - doc.indexorganize 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
> - doc.binary_data 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
> - doc.origdoc 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
> - doc.test_ds16_ds17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119
> - doc.metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
> - doc.ds_28 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
> - doc.sourcebuckets 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
> - doc.ds_18 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
> - doc.sourcecorrelationcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
> - doc.ds_17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119
> - doc.sourcecorrelation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,579
> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
> last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
> latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:65
> - Pool Name Active Pending Completed
> Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
> - ReadStage 0 0 4729901 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
> - CompactionExecutor 0 0 384171
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
> - MutationStage 0 0 14540850 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
> - MemtableReclaimMemory 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
> - PendingRangeCalculator 0 0 11
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69
> - GossipStage 0 0 1126051 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
> - SecondaryIndexManagement 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
> - HintsDispatcher 0 0 15 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
> - Native-Transport-Requests 0 0 13286521
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
> - RequestResponseStage 0 0 15724799
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
> - MemtableFlushWriter 0 0 298
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
> - PerDiskMemtableFlushWriter_0 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
> - MemtablePostFlush 0 0 336 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
> - Sampler 0 0 0 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69
> - ValidationExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69
> - ViewBuildExecutor 0 0 0 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69
> - CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:79
> - CompactionManager 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:91
> - MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:101
> - Cache Type Size Capacity KeysToSave
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:103
> - KeyCache 75540688 104857600 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:109
> - RowCache 0 0 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:116
> - Table Memtable ops,data
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119
> - system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119
> - system_schema.types 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119
> - system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119
> - system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
> - system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
> - system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
> - system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
> - system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
> - system_schema.views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
> - system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
> - system.compaction_history 2,403
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
> - system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119
> - system.repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
> - system.size_estimates 1625,31346
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
> - system.table_estimates 3250,67852
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
> - system.paxos 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
> - system.built_views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
> - system.peer_events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
> - system.peers_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
> - system.peers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
> - system.peer_events_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119
> - system.batches 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
> - system.transferred_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
> - system.transferred_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
> - system.view_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
> - system.local 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
> - system.sstable_activity 508,8564
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
> - system.available_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
> - system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
> - system.prepared_statements 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119
> - system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
> - system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
> - system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
> - system_auth.network_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
> - system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
> - system_distributed.parent_repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
> - system_distributed.repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
> - system_distributed.view_build_status 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
> - reaper_db.node_operations 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119
> - reaper_db.repair_run 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
> - reaper_db.repair_run_by_unit 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
> - reaper_db.cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
> - reaper_db.diagnostic_event_subscription 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
> - reaper_db.running_reapers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
> - reaper_db.node_metrics_v3 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
> - reaper_db.repair_schedule_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
> - reaper_db.repair_unit_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
> - reaper_db.repair_run_by_cluster 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119
> - reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
> - reaper_db.leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
> - reaper_db.schema_migration_leader 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
> - reaper_db.running_repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
> - reaper_db.schema_migration 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
> - reaper_db.node_metrics_v1 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
> - reaper_db.repair_run_by_cluster_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
> - reaper_db.snapshot 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
> - system_traces.sessions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119
> - system_traces.events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
> - doc.doc 71523,9515801
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
> - doc.fieldcounts 719639,2487714
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
> - doc.meta_ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
> - doc.parent_child 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
> - doc.blocks 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
> - doc.ds_30 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
> - doc.child_parent 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
> - doc.seq 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
> - doc.doctemp 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119
> - doc.ds_21 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
> - doc.extractedmetadata 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
> - doc.datasources 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
> - doc.ds_16 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
> - doc.ordered_fieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
> - doc.ds_22 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
> - doc.correlationbuckets 39427,8388316
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
> - doc.ftametrics 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
> - doc.binary_docs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119
> - doc.ordered_metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
> - doc.ds_27 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
> - doc.correlationdocbuckets 450171,39061883
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
> - doc.ds_29 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
> - doc.ds_32 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
> - doc.ds_31 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
> - doc.correlation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
> - doc.ds_20 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
> - doc.indexorganize 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
> - doc.binary_data 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119
> - doc.origdoc 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
> - doc.test_ds16_ds17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
> - doc.metafieldcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
> - doc.ds_28 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
> - doc.sourcebuckets 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
> - doc.ds_18 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
> - doc.sourcecorrelationcounts 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
> - doc.ds_17 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119
> - doc.sourcecorrelation 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594
> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
> last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
> latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:65
> - Pool Name Active Pending Completed
> Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
> - ReadStage 0 0 4730049 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
> - CompactionExecutor 0 0 384171
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
> - MutationStage 0 0 14541404 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
> - MemtableReclaimMemory 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
> - PendingRangeCalculator 0 0 11
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
> - GossipStage 0 0 1126070 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69
> - SecondaryIndexManagement 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
> - HintsDispatcher 0 0 15 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
> - Native-Transport-Requests 0 0 13286967
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
> - RequestResponseStage 0 0 15725284
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
> - MemtableFlushWriter 0 0 298
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
> - PerDiskMemtableFlushWriter_0 0 0 316
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
> - MemtablePostFlush 0 0 336 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
> - Sampler 0 0 0 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
> - ValidationExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
> - ViewBuildExecutor 0 0 0 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69
> - CacheCleanupExecutor 0 0 0
> 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:79
> - CompactionManager 0 0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:91
> - MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:101
> - Cache Type Size Capacity KeysToSave
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:103
> - KeyCache 75542896 104857600 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:109
> - RowCache 0 0 all
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:116
> - Table Memtable ops,data
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119
> - system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119
> - system_schema.types 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119
> - system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119
> - system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system_schema.views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system.compaction_history 2,403
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system.repairs 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system.size_estimates 1625,31346
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system.table_estimates 3250,67852
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119
> - system.paxos 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.built_views 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.peer_events 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.peers_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.peers 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.peer_events_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.batches 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.transferred_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.transferred_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.view_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.local 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.sstable_activity 508,8564
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.available_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119
> - system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
> - system.prepared_statements 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
> - system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
> - system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
> - system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
> - system_auth.network_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
> - system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
> - system_distributed.parent_repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
> - system_distributed.repair_history 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
> - system_distributed.view_build_status 0,0
> INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119
> - reaper_db.node_operations 0,0
> .
> .
> .
>
> and this on the other:
>
> DEBUG [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,174
> InitialConnectionHandler.java:153 - Configured pipeline:
> DefaultChannelPipeline{(frameDecoder =
> org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
> org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
> org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
> org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
> WARN [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,180
> ExceptionHandlers.java:104 - Unknown exception in client networking
> io.netty.channel.unix.Errors$NativeIoException: readAddress(..)
> failed: Connection reset by peer
> DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,275
> InitialConnectionHandler.java:121 - Response to STARTUP sent,
> configuring pipeline for 5/v5
> DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,276
> InitialConnectionHandler.java:153 - Configured pipeline:
> DefaultChannelPipeline{(frameDecoder =
> org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
> org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
> org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
> org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
> WARN [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,278
> ExceptionHandlers.java:104 - Unknown exception in client networking
> io.netty.channel.unix.Errors$NativeIoException: readAddress(..)
> failed: Connection reset by peer
> INFO [Messaging-EventLoop-3-39] 2021-09-10 11:18:41,326
> InboundConnectionInitiator.java:464 -
> /172.16.100.44:7000(/172.16.100.44:44368)->/172.16.100.37:7000-URGENT_MESSAGES-8aa82849
> messaging connection established, version = 12, framing = LZ4,
> encryption = unencrypted
> INFO [ScheduledTasks:1] 2021-09-10 11:18:42,653
> MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
> last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
> latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
> .
> .
> .
>
> -Joe
>
>
> On 9/10/2021 11:06 AM, Sam Tunnicliffe wrote:
>> Is there anything in the logs of the other nodes, particularly those
>> in the seeds list of the one which won't start? If this is a problem
>> with the peers being unable to respond, as the issue fixed in 4.0.1
>> (CASSANDRA-16877) was, you may see some indication there.
>>
>> Sam
>>
>>> On 10 Sep 2021, at 15:56, Joe Obernberger
>>> <jo...@gmail.com> wrote:
>>>
>>> Thank you Jeff - yes, this is on the latest 4.0.1
>>>
>>> nodetool version
>>> ReleaseVersion: 4.0.1
>>> nodetool status
>>> Datacenter: datacenter1
>>> =======================
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> -- Address Load Tokens Owns (effective) Host ID Rack
>>> UN 172.16.100.251 488.38 GiB 200 38.7%
>>> 660f476c-a124-4ca0-b55f-75efe56370da rack1
>>> UN 172.16.100.208 76.02 GiB 30 5.8%
>>> 2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
>>> UN 172.16.100.252 501.88 GiB 200 38.6%
>>> e83aa851-69b4-478f-88f6-60e657ea6539 rack1
>>> UN 172.16.100.249 517.27 GiB 200 38.6%
>>> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
>>> UN 172.16.100.36 524.45 GiB 200 38.6%
>>> d9702f96-256e-45ae-8e12-69a42712be50 rack1
>>> UN 172.16.100.39 521.05 GiB 200 38.6%
>>> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
>>> UN 172.16.100.253 11.39 GiB 4 0.8%
>>> a1a16910-9167-4174-b34b-eb859d36347e rack1
>>> UN 172.16.100.248 524.46 GiB 200 38.7%
>>> 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
>>> UN 172.16.100.37 314.67 GiB 120 23.2%
>>> 08a19658-40be-4e55-8709-812b3d4ac750 rack1
>>> UN 172.16.100.250 464.23 GiB 200 38.6%
>>> b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>>>
>>> yum list installed | grep cass
>>> cassandra.noarch 4.0.1-1 @cassandra
>>>
>>> -Joe
>>>
>>> On 9/10/2021 10:54 AM, Jeff Jirsa wrote:
>>>> Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is
>>>> too large for the urgent message queue, causing this stack trace,
>>>> and was released 3 days ago. I've never seen it on a 10 node
>>>> cluster before, but I'd be trying that.
>>>>
>>>> On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger
>>>> <jo...@gmail.com> wrote:
>>>>
>>>> I have a 10 node cluster and am trying to add another node. The
>>>> new
>>>> node is running Rocky Linux and I'm getting the unable to
>>>> gossip with
>>>> any peers error. Firewall and SELinux are off. I can ping all the
>>>> other nodes OK. I've checked everything I can think of
>>>> (/etc/hosts,
>>>> listen_address, broadcast etc..). It all looks correct to me.
>>>> Any ideas? Could it be an incompatibility with Rocky?
>>>>
>>>> DEBUG [main] 2021-09-10 06:45:24,846
>>>> YamlConfigurationLoader.java:112 -
>>>> Loading settings from
>>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>>> INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
>>>> OutboundConnection.java:1150 -
>>>> /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.
>>>> <http://172.16.0.100/>253:7000-URGENT_MESSAGES-90efbb9e
>>>> successfully connected, version = 12, framing = LZ4, encryption =
>>>> unencrypted
>>>> INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
>>>> OutboundConnection.java:1150 -
>>>> /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37
>>>> <http://172.16.100.37/>:7000-URGENT_MESSAGES-eae47864
>>>> successfully connected, version = 12, framing = LZ4, encryption =
>>>> unencrypted
>>>> INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648
>>>> TokenMetadata.java:525
>>>> - Updating topology for all endpoints that have changed
>>>> DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
>>>> SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
>>>> recording size estimates
>>>> ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
>>>> Exception encountered during startup
>>>> java.lang.RuntimeException: Unable to gossip with any peers
>>>> at
>>>> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
>>>> at
>>>> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
>>>> at
>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
>>>> at
>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
>>>> at
>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
>>>> DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
>>>> StorageService.java:1621 - DRAINING: starting drain process
>>>> INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
>>>> HintsService.java:220 - Paused hints dispatch
>>>> WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
>>>> Gossiper.java:1993 - No local state, state is in silent
>>>> shutdown, or
>>>> node hasn't joined, not announcing shutdown
>>>>
>>>> Thank you!
>>>>
>>>> -Joe
>>>>
>>>>
>>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>>> Virus-free. www.avg.com
>>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>>>
>>>>
>>>> <x-msg://5/#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
Re: Unable to Gossip
Posted by Joe Obernberger <jo...@gmail.com>.
Good idea.
There are two seed nodes:
I see this on one (note 172.16.100.44 is the new node):
DEBUG [CompactionExecutor:1345] 2021-09-10 11:13:49,569
TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully
expired SSTables
INFO [Messaging-EventLoop-3-10] 2021-09-10 11:14:22,810
InboundConnectionInitiator.java:464 -
/172.16.100.44:7000(/172.16.100.44:45970)->/172.16.100.253:7000-URGENT_MESSAGES-30a4fd82
messaging connection established, version = 12, framing = LZ4,
encryption = unencrypted
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:65 -
Pool Name Active Pending Completed Blocked
All Time Blocked
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69 -
ReadStage 0 0 4729810 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69 -
CompactionExecutor 0 0 384171 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,567 StatusLogger.java:69 -
MutationStage 0 0 14540487 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
MemtableReclaimMemory 0 0 316 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
PendingRangeCalculator 0 0 11
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
GossipStage 0 0 1126031 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
SecondaryIndexManagement 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
HintsDispatcher 0 0 15 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
Native-Transport-Requests 0 0 13286230
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
RequestResponseStage 0 0 15724485 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
MemtableFlushWriter 0 0 298 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
PerDiskMemtableFlushWriter_0 0 0 316
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,568 StatusLogger.java:69 -
MemtablePostFlush 0 0 336 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
ValidationExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:69 -
CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,569 StatusLogger.java:79 -
CompactionManager 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:91 -
MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:101 -
Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:103 -
KeyCache 75539240 104857600 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:109 -
RowCache 0 0 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:116 -
Table Memtable ops,data
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
system_schema.columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
system_schema.types 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
system_schema.indexes 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
system_schema.keyspaces 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,570 StatusLogger.java:119 -
system_schema.dropped_columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
system_schema.aggregates 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
system_schema.triggers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
system_schema.tables 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
system_schema.views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
system_schema.functions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
system.compaction_history 2,403
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
system.repairs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
system.size_estimates 1625,31346
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,571 StatusLogger.java:119 -
system.table_estimates 3250,67852
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.paxos 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.built_views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.peer_events 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.peers_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.peers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.peer_events_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.batches 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.transferred_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.transferred_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.view_builds_in_progress 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,572 StatusLogger.java:119 -
system.local 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
system.sstable_activity 508,8564
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
system.available_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
system.available_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
system.prepared_statements 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
system_auth.roles 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
system_auth.role_members 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
system_auth.resource_role_permissons_index 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
system_auth.network_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
system_auth.role_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,573 StatusLogger.java:119 -
system_distributed.parent_repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
system_distributed.repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
system_distributed.view_build_status 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
reaper_db.node_operations 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
reaper_db.repair_run 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
reaper_db.repair_run_by_unit 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
reaper_db.cluster 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
reaper_db.diagnostic_event_subscription 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
reaper_db.running_reapers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
reaper_db.node_metrics_v3 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
reaper_db.repair_schedule_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,574 StatusLogger.java:119 -
reaper_db.repair_unit_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
reaper_db.repair_run_by_cluster 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
reaper_db.leader 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
reaper_db.schema_migration_leader 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
reaper_db.running_repairs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
reaper_db.schema_migration 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
reaper_db.node_metrics_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
reaper_db.repair_run_by_cluster_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
reaper_db.snapshot 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,575 StatusLogger.java:119 -
system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
system_traces.events 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
doc.doc 71463,9507451
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
doc.fieldcounts 719515,2487290
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
doc.meta_ftametrics 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
doc.parent_child 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
doc.blocks 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
doc.ds_30 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
doc.child_parent 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
doc.seq 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
doc.doctemp 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,576 StatusLogger.java:119 -
doc.ds_21 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.extractedmetadata 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.datasources 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.ds_16 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.ordered_fieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.ds_22 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.correlationbuckets 39374,8377095
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.ftametrics 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.binary_docs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.ordered_metafieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.ds_27 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,577 StatusLogger.java:119 -
doc.correlationdocbuckets 450045,39051043
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
doc.ds_29 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
doc.ds_32 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
doc.ds_31 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
doc.correlation 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
doc.ds_20 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
doc.indexorganize 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
doc.binary_data 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
doc.origdoc 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
doc.test_ds16_ds17 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,578 StatusLogger.java:119 -
doc.metafieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
doc.ds_28 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
doc.sourcebuckets 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
doc.ds_18 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
doc.sourcecorrelationcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
doc.ds_17 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:26,579 StatusLogger.java:119 -
doc.sourcecorrelation 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,579
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:65 -
Pool Name Active Pending Completed Blocked
All Time Blocked
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
ReadStage 0 0 4729901 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
CompactionExecutor 0 0 384171 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
MutationStage 0 0 14540850 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
MemtableReclaimMemory 0 0 316 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
PendingRangeCalculator 0 0 11
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,580 StatusLogger.java:69 -
GossipStage 0 0 1126051 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
SecondaryIndexManagement 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
HintsDispatcher 0 0 15 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
Native-Transport-Requests 0 0 13286521
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
RequestResponseStage 0 0 15724799 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
MemtableFlushWriter 0 0 298 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
PerDiskMemtableFlushWriter_0 0 0 316
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
MemtablePostFlush 0 0 336 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,581 StatusLogger.java:69 -
ValidationExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69 -
ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:69 -
CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:79 -
CompactionManager 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,582 StatusLogger.java:91 -
MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:101 -
Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:103 -
KeyCache 75540688 104857600 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:109 -
RowCache 0 0 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:116 -
Table Memtable ops,data
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
system_schema.columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
system_schema.types 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
system_schema.indexes 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,583 StatusLogger.java:119 -
system_schema.keyspaces 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
system_schema.dropped_columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
system_schema.aggregates 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
system_schema.triggers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
system_schema.tables 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
system_schema.views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
system_schema.functions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
system.compaction_history 2,403
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,584 StatusLogger.java:119 -
system.repairs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
system.size_estimates 1625,31346
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
system.table_estimates 3250,67852
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
system.paxos 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
system.built_views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
system.peer_events 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
system.peers_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
system.peers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
system.peer_events_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,585 StatusLogger.java:119 -
system.batches 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
system.transferred_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
system.transferred_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
system.view_builds_in_progress 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
system.local 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
system.sstable_activity 508,8564
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
system.available_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
system.available_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
system.prepared_statements 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,586 StatusLogger.java:119 -
system_auth.roles 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
system_auth.role_members 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
system_auth.resource_role_permissons_index 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
system_auth.network_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
system_auth.role_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
system_distributed.parent_repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
system_distributed.repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
system_distributed.view_build_status 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
reaper_db.node_operations 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,587 StatusLogger.java:119 -
reaper_db.repair_run 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
reaper_db.repair_run_by_unit 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
reaper_db.cluster 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
reaper_db.diagnostic_event_subscription 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
reaper_db.running_reapers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
reaper_db.node_metrics_v3 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
reaper_db.repair_schedule_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
reaper_db.repair_unit_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
reaper_db.repair_run_by_cluster 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,588 StatusLogger.java:119 -
reaper_db.repair_schedule_by_cluster_and_keyspace 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
reaper_db.leader 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
reaper_db.schema_migration_leader 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
reaper_db.running_repairs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
reaper_db.schema_migration 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
reaper_db.node_metrics_v1 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
reaper_db.repair_run_by_cluster_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
reaper_db.snapshot 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,589 StatusLogger.java:119 -
system_traces.events 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
doc.doc 71523,9515801
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
doc.fieldcounts 719639,2487714
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
doc.meta_ftametrics 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
doc.parent_child 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
doc.blocks 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
doc.ds_30 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
doc.child_parent 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
doc.seq 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
doc.doctemp 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,590 StatusLogger.java:119 -
doc.ds_21 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
doc.extractedmetadata 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
doc.datasources 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
doc.ds_16 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
doc.ordered_fieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
doc.ds_22 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
doc.correlationbuckets 39427,8388316
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
doc.ftametrics 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
doc.binary_docs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,591 StatusLogger.java:119 -
doc.ordered_metafieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
doc.ds_27 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
doc.correlationdocbuckets 450171,39061883
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
doc.ds_29 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
doc.ds_32 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
doc.ds_31 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
doc.correlation 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
doc.ds_20 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
doc.indexorganize 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
doc.binary_data 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,592 StatusLogger.java:119 -
doc.origdoc 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
doc.test_ds16_ds17 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
doc.metafieldcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
doc.ds_28 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
doc.sourcebuckets 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
doc.ds_18 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
doc.sourcecorrelationcounts 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
doc.ds_17 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:31,593 StatusLogger.java:119 -
doc.sourcecorrelation 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:65 -
Pool Name Active Pending Completed Blocked
All Time Blocked
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
ReadStage 0 0 4730049 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
CompactionExecutor 0 0 384171 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
MutationStage 0 0 14541404 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
MemtableReclaimMemory 0 0 316 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
PendingRangeCalculator 0 0 11
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
GossipStage 0 0 1126070 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,594 StatusLogger.java:69 -
SecondaryIndexManagement 0 0 0
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
HintsDispatcher 0 0 15 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
Native-Transport-Requests 0 0 13286967
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
RequestResponseStage 0 0 15725284 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
MemtableFlushWriter 0 0 298 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
PerDiskMemtableFlushWriter_0 0 0 316
0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
MemtablePostFlush 0 0 336 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
ValidationExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,595 StatusLogger.java:69 -
CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:79 -
CompactionManager 0 0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:91 -
MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:101 -
Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:103 -
KeyCache 75542896 104857600 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:109 -
RowCache 0 0 all
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:116 -
Table Memtable ops,data
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
system_schema.columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
system_schema.types 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
system_schema.indexes 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,596 StatusLogger.java:119 -
system_schema.keyspaces 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system_schema.dropped_columns 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system_schema.aggregates 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system_schema.triggers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system_schema.tables 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system_schema.views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system_schema.functions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system.compaction_history 2,403
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system.repairs 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system.size_estimates 1625,31346
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system.table_estimates 3250,67852
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,597 StatusLogger.java:119 -
system.paxos 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.built_views 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.peer_events 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.peers_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.peers 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.peer_events_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.batches 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.transferred_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.transferred_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.view_builds_in_progress 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.local 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.sstable_activity 508,8564
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.available_ranges_v2 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,598 StatusLogger.java:119 -
system.available_ranges 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
system.prepared_statements 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
system_auth.roles 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
system_auth.role_members 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
system_auth.resource_role_permissons_index 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
system_auth.network_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
system_auth.role_permissions 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
system_distributed.parent_repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
system_distributed.repair_history 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
system_distributed.view_build_status 0,0
INFO [ScheduledTasks:1] 2021-09-10 11:14:36,599 StatusLogger.java:119 -
reaper_db.node_operations 0,0
.
.
.
and this on the other:
DEBUG [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,174
InitialConnectionHandler.java:153 - Configured pipeline:
DefaultChannelPipeline{(frameDecoder =
org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
WARN [epollEventLoopGroup-5-77] 2021-09-10 11:18:35,180
ExceptionHandlers.java:104 - Unknown exception in client networking
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer
DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,275
InitialConnectionHandler.java:121 - Response to STARTUP sent,
configuring pipeline for 5/v5
DEBUG [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,276
InitialConnectionHandler.java:153 - Configured pipeline:
DefaultChannelPipeline{(frameDecoder =
org.apache.cassandra.net.FrameDecoderCrc), (frameEncoder =
org.apache.cassandra.net.FrameEncoderCrc), (cqlProcessor =
org.apache.cassandra.transport.CQLMessageHandler), (exceptionHandler =
org.apache.cassandra.transport.ExceptionHandlers$PostV5ExceptionHandler)}
WARN [epollEventLoopGroup-5-78] 2021-09-10 11:18:36,278
ExceptionHandlers.java:104 - Unknown exception in client networking
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer
INFO [Messaging-EventLoop-3-39] 2021-09-10 11:18:41,326
InboundConnectionInitiator.java:464 -
/172.16.100.44:7000(/172.16.100.44:44368)->/172.16.100.37:7000-URGENT_MESSAGES-8aa82849
messaging connection established, version = 12, framing = LZ4,
encryption = unencrypted
INFO [ScheduledTasks:1] 2021-09-10 11:18:42,653
MessagingMetrics.java:206 - GOSSIP_DIGEST_SYN messages were dropped in
last 5000 ms: 0 internal and 1 cross node. Mean internal dropped
latency: 0 ms and Mean cross-node dropped latency: 15137813 ms
.
.
.
-Joe
On 9/10/2021 11:06 AM, Sam Tunnicliffe wrote:
> Is there anything in the logs of the other nodes, particularly those
> in the seeds list of the one which won't start? If this is a problem
> with the peers being unable to respond, as the issue fixed in 4.0.1
> (CASSANDRA-16877) was, you may see some indication there.
>
> Sam
>
>> On 10 Sep 2021, at 15:56, Joe Obernberger
>> <jo...@gmail.com> wrote:
>>
>> Thank you Jeff - yes, this is on the latest 4.0.1
>>
>> nodetool version
>> ReleaseVersion: 4.0.1
>> nodetool status
>> Datacenter: datacenter1
>> =======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID Rack
>> UN 172.16.100.251 488.38 GiB 200 38.7%
>> 660f476c-a124-4ca0-b55f-75efe56370da rack1
>> UN 172.16.100.208 76.02 GiB 30 5.8%
>> 2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
>> UN 172.16.100.252 501.88 GiB 200 38.6%
>> e83aa851-69b4-478f-88f6-60e657ea6539 rack1
>> UN 172.16.100.249 517.27 GiB 200 38.6%
>> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
>> UN 172.16.100.36 524.45 GiB 200 38.6%
>> d9702f96-256e-45ae-8e12-69a42712be50 rack1
>> UN 172.16.100.39 521.05 GiB 200 38.6%
>> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
>> UN 172.16.100.253 11.39 GiB 4 0.8%
>> a1a16910-9167-4174-b34b-eb859d36347e rack1
>> UN 172.16.100.248 524.46 GiB 200 38.7%
>> 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
>> UN 172.16.100.37 314.67 GiB 120 23.2%
>> 08a19658-40be-4e55-8709-812b3d4ac750 rack1
>> UN 172.16.100.250 464.23 GiB 200 38.6%
>> b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>>
>> yum list installed | grep cass
>> cassandra.noarch 4.0.1-1 @cassandra
>>
>> -Joe
>>
>> On 9/10/2021 10:54 AM, Jeff Jirsa wrote:
>>> Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is
>>> too large for the urgent message queue, causing this stack trace,
>>> and was released 3 days ago. I've never seen it on a 10 node cluster
>>> before, but I'd be trying that.
>>>
>>> On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger
>>> <jo...@gmail.com> wrote:
>>>
>>> I have a 10 node cluster and am trying to add another node. The new
>>> node is running Rocky Linux and I'm getting the unable to gossip
>>> with
>>> any peers error. Firewall and SELinux are off. I can ping all the
>>> other nodes OK. I've checked everything I can think of
>>> (/etc/hosts,
>>> listen_address, broadcast etc..). It all looks correct to me.
>>> Any ideas? Could it be an incompatibility with Rocky?
>>>
>>> DEBUG [main] 2021-09-10 06:45:24,846
>>> YamlConfigurationLoader.java:112 -
>>> Loading settings from
>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>> INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
>>> OutboundConnection.java:1150 -
>>> /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.
>>> <http://172.16.0.100/>253:7000-URGENT_MESSAGES-90efbb9e
>>> successfully connected, version = 12, framing = LZ4, encryption =
>>> unencrypted
>>> INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
>>> OutboundConnection.java:1150 -
>>> /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37
>>> <http://172.16.100.37/>:7000-URGENT_MESSAGES-eae47864
>>> successfully connected, version = 12, framing = LZ4, encryption =
>>> unencrypted
>>> INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648
>>> TokenMetadata.java:525
>>> - Updating topology for all endpoints that have changed
>>> DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
>>> SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
>>> recording size estimates
>>> ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
>>> Exception encountered during startup
>>> java.lang.RuntimeException: Unable to gossip with any peers
>>> at
>>> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
>>> at
>>> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
>>> at
>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
>>> DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
>>> StorageService.java:1621 - DRAINING: starting drain process
>>> INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
>>> HintsService.java:220 - Paused hints dispatch
>>> WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
>>> Gossiper.java:1993 - No local state, state is in silent
>>> shutdown, or
>>> node hasn't joined, not announcing shutdown
>>>
>>> Thank you!
>>>
>>> -Joe
>>>
>>>
>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>> Virus-free. www.avg.com
>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>>
>>>
>>> <x-msg://5/#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
Re: Unable to Gossip
Posted by Sam Tunnicliffe <sa...@beobal.com>.
Is there anything in the logs of the other nodes, particularly those in the seeds list of the one which won't start? If this is a problem with the peers being unable to respond, as the issue fixed in 4.0.1 (CASSANDRA-16877) was, you may see some indication there.
Sam
> On 10 Sep 2021, at 15:56, Joe Obernberger <jo...@gmail.com> wrote:
>
> Thank you Jeff - yes, this is on the latest 4.0.1
>
> nodetool version
> ReleaseVersion: 4.0.1
> nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 172.16.100.251 488.38 GiB 200 38.7% 660f476c-a124-4ca0-b55f-75efe56370da rack1
> UN 172.16.100.208 76.02 GiB 30 5.8% 2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
> UN 172.16.100.252 501.88 GiB 200 38.6% e83aa851-69b4-478f-88f6-60e657ea6539 rack1
> UN 172.16.100.249 517.27 GiB 200 38.6% 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
> UN 172.16.100.36 524.45 GiB 200 38.6% d9702f96-256e-45ae-8e12-69a42712be50 rack1
> UN 172.16.100.39 521.05 GiB 200 38.6% 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
> UN 172.16.100.253 11.39 GiB 4 0.8% a1a16910-9167-4174-b34b-eb859d36347e rack1
> UN 172.16.100.248 524.46 GiB 200 38.7% 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
> UN 172.16.100.37 314.67 GiB 120 23.2% 08a19658-40be-4e55-8709-812b3d4ac750 rack1
> UN 172.16.100.250 464.23 GiB 200 38.6% b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>
> yum list installed | grep cass
> cassandra.noarch 4.0.1-1 @cassandra
>
> -Joe
>
> On 9/10/2021 10:54 AM, Jeff Jirsa wrote:
>> Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is too large for the urgent message queue, causing this stack trace, and was released 3 days ago. I've never seen it on a 10 node cluster before, but I'd be trying that.
>>
>> On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger <joseph.obernberger@gmail.com <ma...@gmail.com>> wrote:
>> I have a 10 node cluster and am trying to add another node. The new
>> node is running Rocky Linux and I'm getting the unable to gossip with
>> any peers error. Firewall and SELinux are off. I can ping all the
>> other nodes OK. I've checked everything I can think of (/etc/hosts,
>> listen_address, broadcast etc..). It all looks correct to me.
>> Any ideas? Could it be an incompatibility with Rocky?
>>
>> DEBUG [main] 2021-09-10 06:45:24,846 YamlConfigurationLoader.java:112 -
>> Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml <file:///etc/cassandra/default.conf/cassandra.yaml>
>> INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
>> OutboundConnection.java:1150 -
>> /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100. <http://172.16.0.100/>253:7000-URGENT_MESSAGES-90efbb9e
>> successfully connected, version = 12, framing = LZ4, encryption =
>> unencrypted
>> INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
>> OutboundConnection.java:1150 -
>> /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37 <http://172.16.100.37/>:7000-URGENT_MESSAGES-eae47864
>> successfully connected, version = 12, framing = LZ4, encryption =
>> unencrypted
>> INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648 TokenMetadata.java:525
>> - Updating topology for all endpoints that have changed
>> DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
>> SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
>> recording size estimates
>> ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
>> Exception encountered during startup
>> java.lang.RuntimeException: Unable to gossip with any peers
>> at
>> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
>> at
>> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
>> at
>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
>> DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
>> StorageService.java:1621 - DRAINING: starting drain process
>> INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
>> HintsService.java:220 - Paused hints dispatch
>> WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
>> Gossiper.java:1993 - No local state, state is in silent shutdown, or
>> node hasn't joined, not announcing shutdown
>>
>> Thank you!
>>
>> -Joe
>>
>>
>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> <x-msg://5/#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Unable to Gossip
Posted by Jeff Jirsa <jj...@gmail.com>.
Can you drop as much info as possible into a JIRA?
Include the output of `nodetool gossipinfo` if at all possible
On Fri, Sep 10, 2021 at 7:58 AM Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> Thank you Jeff - yes, this is on the latest 4.0.1
>
> nodetool version
> ReleaseVersion: 4.0.1
> nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN 172.16.100.251 488.38 GiB 200 38.7%
> 660f476c-a124-4ca0-b55f-75efe56370da rack1
> UN 172.16.100.208 76.02 GiB 30 5.8%
> 2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
> UN 172.16.100.252 501.88 GiB 200 38.6%
> e83aa851-69b4-478f-88f6-60e657ea6539 rack1
> UN 172.16.100.249 517.27 GiB 200 38.6%
> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
> UN 172.16.100.36 524.45 GiB 200 38.6%
> d9702f96-256e-45ae-8e12-69a42712be50 rack1
> UN 172.16.100.39 521.05 GiB 200 38.6%
> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
> UN 172.16.100.253 11.39 GiB 4 0.8%
> a1a16910-9167-4174-b34b-eb859d36347e rack1
> UN 172.16.100.248 524.46 GiB 200 38.7%
> 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
> UN 172.16.100.37 314.67 GiB 120 23.2%
> 08a19658-40be-4e55-8709-812b3d4ac750 rack1
> UN 172.16.100.250 464.23 GiB 200 38.6%
> b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>
> yum list installed | grep cass
> cassandra.noarch 4.0.1-1
> @cassandra
>
> -Joe
> On 9/10/2021 10:54 AM, Jeff Jirsa wrote:
>
> Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is too
> large for the urgent message queue, causing this stack trace, and was
> released 3 days ago. I've never seen it on a 10 node cluster before, but
> I'd be trying that.
>
> On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger <
> joseph.obernberger@gmail.com> wrote:
>
>> I have a 10 node cluster and am trying to add another node. The new
>> node is running Rocky Linux and I'm getting the unable to gossip with
>> any peers error. Firewall and SELinux are off. I can ping all the
>> other nodes OK. I've checked everything I can think of (/etc/hosts,
>> listen_address, broadcast etc..). It all looks correct to me.
>> Any ideas? Could it be an incompatibility with Rocky?
>>
>> DEBUG [main] 2021-09-10 06:45:24,846 YamlConfigurationLoader.java:112 -
>> Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
>> INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
>> OutboundConnection.java:1150 -
>> /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.253:7000-URGENT_MESSAGES-90efbb9e
>>
>> successfully connected, version = 12, framing = LZ4, encryption =
>> unencrypted
>> INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
>> OutboundConnection.java:1150 -
>> /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37:7000-URGENT_MESSAGES-eae47864
>>
>> successfully connected, version = 12, framing = LZ4, encryption =
>> unencrypted
>> INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648 TokenMetadata.java:525
>> - Updating topology for all endpoints that have changed
>> DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
>> SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
>> recording size estimates
>> ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
>> Exception encountered during startup
>> java.lang.RuntimeException: Unable to gossip with any peers
>> at
>> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
>> at
>>
>> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
>> at
>>
>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
>> at
>>
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
>> at
>>
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
>> at
>>
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
>> at
>>
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
>> at
>>
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
>> DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
>> StorageService.java:1621 - DRAINING: starting drain process
>> INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
>> HintsService.java:220 - Paused hints dispatch
>> WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
>> Gossiper.java:1993 - No local state, state is in silent shutdown, or
>> node hasn't joined, not announcing shutdown
>>
>> Thank you!
>>
>> -Joe
>>
>>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> <#m_-4486529990198249803_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
>
Re: Unable to Gossip
Posted by Joe Obernberger <jo...@gmail.com>.
Thank you Jeff - yes, this is on the latest 4.0.1
nodetool version
ReleaseVersion: 4.0.1
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN 172.16.100.251 488.38 GiB 200 38.7%
660f476c-a124-4ca0-b55f-75efe56370da rack1
UN 172.16.100.208 76.02 GiB 30 5.8%
2529b6ed-cdb2-43c2-bdd7-171cfe308bd3 rack1
UN 172.16.100.252 501.88 GiB 200 38.6%
e83aa851-69b4-478f-88f6-60e657ea6539 rack1
UN 172.16.100.249 517.27 GiB 200 38.6%
49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
UN 172.16.100.36 524.45 GiB 200 38.6%
d9702f96-256e-45ae-8e12-69a42712be50 rack1
UN 172.16.100.39 521.05 GiB 200 38.6%
93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
UN 172.16.100.253 11.39 GiB 4 0.8%
a1a16910-9167-4174-b34b-eb859d36347e rack1
UN 172.16.100.248 524.46 GiB 200 38.7%
4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
UN 172.16.100.37 314.67 GiB 120 23.2%
08a19658-40be-4e55-8709-812b3d4ac750 rack1
UN 172.16.100.250 464.23 GiB 200 38.6%
b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
yum list installed | grep cass
cassandra.noarch 4.0.1-1 @cassandra
-Joe
On 9/10/2021 10:54 AM, Jeff Jirsa wrote:
> Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is too
> large for the urgent message queue, causing this stack trace, and was
> released 3 days ago. I've never seen it on a 10 node cluster before,
> but I'd be trying that.
>
> On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger
> <jo...@gmail.com> wrote:
>
> I have a 10 node cluster and am trying to add another node. The new
> node is running Rocky Linux and I'm getting the unable to gossip with
> any peers error. Firewall and SELinux are off. I can ping all the
> other nodes OK. I've checked everything I can think of (/etc/hosts,
> listen_address, broadcast etc..). It all looks correct to me.
> Any ideas? Could it be an incompatibility with Rocky?
>
> DEBUG [main] 2021-09-10 06:45:24,846
> YamlConfigurationLoader.java:112 -
> Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
> INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
> OutboundConnection.java:1150 -
> /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.
> <http://172.16.100.>253:7000-URGENT_MESSAGES-90efbb9e
> successfully connected, version = 12, framing = LZ4, encryption =
> unencrypted
> INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
> OutboundConnection.java:1150 -
> /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37
> <http://172.16.100.37>:7000-URGENT_MESSAGES-eae47864
> successfully connected, version = 12, framing = LZ4, encryption =
> unencrypted
> INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648
> TokenMetadata.java:525
> - Updating topology for all endpoints that have changed
> DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
> SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
> recording size estimates
> ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
> Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any peers
> at
> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
> at
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
> at
> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
> DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
> StorageService.java:1621 - DRAINING: starting drain process
> INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
> HintsService.java:220 - Paused hints dispatch
> WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
> Gossiper.java:1993 - No local state, state is in silent shutdown, or
> node hasn't joined, not announcing shutdown
>
> Thank you!
>
> -Joe
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> Virus-free. www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Unable to Gossip
Posted by Jeff Jirsa <jj...@gmail.com>.
Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is too
large for the urgent message queue, causing this stack trace, and was
released 3 days ago. I've never seen it on a 10 node cluster before, but
I'd be trying that.
On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> I have a 10 node cluster and am trying to add another node. The new
> node is running Rocky Linux and I'm getting the unable to gossip with
> any peers error. Firewall and SELinux are off. I can ping all the
> other nodes OK. I've checked everything I can think of (/etc/hosts,
> listen_address, broadcast etc..). It all looks correct to me.
> Any ideas? Could it be an incompatibility with Rocky?
>
> DEBUG [main] 2021-09-10 06:45:24,846 YamlConfigurationLoader.java:112 -
> Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml
> INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921
> OutboundConnection.java:1150 -
> /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.253:7000-URGENT_MESSAGES-90efbb9e
>
> successfully connected, version = 12, framing = LZ4, encryption =
> unencrypted
> INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930
> OutboundConnection.java:1150 -
> /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37:7000-URGENT_MESSAGES-eae47864
>
> successfully connected, version = 12, framing = LZ4, encryption =
> unencrypted
> INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648 TokenMetadata.java:525
> - Updating topology for all endpoints that have changed
> DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644
> SizeEstimatesRecorder.java:65 - Node is not part of the ring; not
> recording size estimates
> ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 -
> Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any peers
> at
> org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805)
> at
>
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648)
> at
>
> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934)
> at
>
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:784)
> at
>
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:729)
> at
>
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
> at
>
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887)
> DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896
> StorageService.java:1621 - DRAINING: starting drain process
> INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898
> HintsService.java:220 - Paused hints dispatch
> WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899
> Gossiper.java:1993 - No local state, state is in silent shutdown, or
> node hasn't joined, not announcing shutdown
>
> Thank you!
>
> -Joe
>
>