You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (Jira)" <ji...@apache.org> on 2022/10/26 17:45:00 UTC
[jira] [Updated] (CASSANDRA-17996) Authentication timed out
[ https://issues.apache.org/jira/browse/CASSANDRA-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-17996:
-----------------------------------------
Resolution: Invalid
Status: Resolved (was: Triage Needed)
This issue tracker is for the development of Apache Cassandra and as such makes a poor vehicle for support. I recommend contacting the community through the mailing or slack for your issue: https://cassandra.apache.org/_/community.html
> Authentication timed out
> ------------------------
>
> Key: CASSANDRA-17996
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17996
> Project: Cassandra
> Issue Type: Bug
> Reporter: Andy Lage
> Priority: Normal
>
> Hi,
>
> I'm having some issues with my authentication when trying to login with cqlsh. I'm running a cluster of 3 nodes that is spaced out on three different physical nodes in Kubernetes. It has been rolling like a charm for the last month but around a week or so, it started to go down. Down below you can see when I try to login to the different nodes and the responses. (a FYI it's not always node 0 that has problem, I've also seen node 1 has the same problem but then node 0 worked fine.
> {code:java}
> I have no name!@cassandra-0:/$ cqlsh -u cassandra -p abc123
> Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': OperationTimedOut('errors=Timed out creating connection (5 seconds), last_host=None',)})
> I have no name!@cassandra-1:/$ cqlsh -u cassandra -p abc123
> Python 2.7 support is deprecated. Install Python 3.6+ or set CQLSH_NO_WARN_PY2 to suppress this message.Connected to mycluster at 127.0.0.1:9042
> [cqlsh 6.0.0 | Cassandra 4.0.0 | CQL spec 3.4.5 | Native protocol v5]
> Use HELP for help.
> cassandra@cqlsh>
> I have no name!@cassandra-2:/$ cqlsh -u cassandra -p abc123
> Python 2.7 support is deprecated. Install Python 3.6+ or set CQLSH_NO_WARN_PY2 to suppress this message.Connected to mycluster at 127.0.0.1:9042
> [cqlsh 6.0.0 | Cassandra 4.0.0 | CQL spec 3.4.5 | Native protocol v5]
> Use HELP for help.
> cassandra@cqlsh>{code}
> Here is the logs from node 0
>
> {code:java}
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 MessagingMetrics.java:206 - HINT_RSP messages were dropped in last 5000 ms: 0 internal and 4 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 8582 ms
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 MessagingMetrics.java:206 - HINT_REQ messages were dropped in last 5000 ms: 0 internal and 2 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 8582 ms
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:65 - Pool Name Active Pending Completed Blocked All Time Blocked
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:69 - ReadStage 0 0 17086 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:69 - CompactionExecutor 0 0 195522 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:69 - MutationStage 0 0 88515 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - MemtableReclaimMemory 0 0 210 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - PendingRangeCalculator 0 0 13 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - GossipStage 0 0 660680 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - SecondaryIndexManagement 0 0 0 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - HintsDispatcher 2 0 37776 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - Repair-Task 0 0 5 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - Native-Transport-Requests 0 0 17111 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - RequestResponseStage 0 0 20 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - MemtableFlushWriter 0 0 210 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0 0 210 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - MemtablePostFlush 0 0 255 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - Sampler 0 0 0 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - ValidationExecutor 0 0 41 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - ViewBuildExecutor 0 0 0 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - InternalResponseStage 0 0 151676 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:69 - AntiEntropyStage 0 0 249 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:69 - CacheCleanupExecutor 0 0 0 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:79 - CompactionManager 0 0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:91 - MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:101 - Cache Type Size Capacity KeysToSave
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:103 - KeyCache 11452 75497472 all
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:109 - RowCache 0 0 all
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:116 - Table Memtable ops,data
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:119 - system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.types 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.views 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.compaction_history 3,634
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.repairs 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.size_estimates 49344,1087744
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.table_estimates 98688,2339296
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.paxos 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.built_views 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.peer_events 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.peers_v2 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.peers 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.peer_events_v2 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.batches 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.transferred_ranges 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.transferred_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.view_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.local 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.sstable_activity 229,2816
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.available_ranges_v2 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.prepared_statements 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - system_auth.network_permissions 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,100 StatusLogger.java:119 - system_distributed.parent_repair_history 10,418916
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,100 StatusLogger.java:119 - system_distributed.repair_history 29495,15967
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,100 StatusLogger.java:119 - system_distributed.view_build_status 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,101 StatusLogger.java:119 - system_traces.sessions 0,0
> INFO [ScheduledTasks:1] 2022-10-26 17:39:00,101 StatusLogger.java:119 - system_traces.events 0,0
> {code}
> The keyspace replication
>
> {code:java}
> cassandra@cqlsh> describe keyspace system_auth;
> CREATE KEYSPACE system_auth WITH replication = {'class':
> 'NetworkTopologyStrategy', 'datacenter1': '3'} AND durable_writes = true;
> {code}
> Nodetool status
> {code:java}
> I have no name!@cassandra-2:/$ nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.233.92.33 3.63 MiB 256 100.0% xxx rack1 UN 10.233.96.184 4.2 MiB 256 100.0% xxx rack1 UN 10.233.90.48 3.24 MiB 256 100.0% xxx rack1 {code}
> I've tried to do nodetool repair without success.
>
> Does anyone have a clue whats going on?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org