You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jai Bheemsen Rao Dhanwada (Jira)" <ji...@apache.org> on 2019/12/12 19:19:00 UTC
[jira] [Created] (CASSANDRA-15449) Credentials out of sync after
replacing the nodes
Jai Bheemsen Rao Dhanwada created CASSANDRA-15449:
-----------------------------------------------------
Summary: Credentials out of sync after replacing the nodes
Key: CASSANDRA-15449
URL: https://issues.apache.org/jira/browse/CASSANDRA-15449
Project: Cassandra
Issue Type: Bug
Reporter: Jai Bheemsen Rao Dhanwada
Attachments: Screen Shot 2019-12-12 at 11.13.52 AM.png
Hello,
We are seeing a strange issue where, after replacing multiple C* nodes from the clusters intermittently we see an issue where few nodes doesn't have any credentials and the client queries fail.
Here are the sequence of steps
1. on a Multi DC C* cluster(12 nodes in each DC), we replaced all the nodes in one DC.
2. The approach we took to replace the nodes is kill one node and launch a new node with {{-Dcassandra.replace_address=}} and proceed with next node once the node is bootstrapped and CQL is enabled.
3. This process works fine and all of a sudden, we started seeing our application started failing with the below errors in the logs
{quote}com.datastax.driver.core.exceptions.UnauthorizedException: User abc has no SELECT permission on <table ks.cf> or any of its parents at com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:59) at com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:25) at
{quote}
4. At this stage we see that 3 nodes in the cluster takes zero traffic, while rest of the nodes are serving ~100 requests. (attached the metrics)
!Screen Shot 2019-12-12 at 11.13.52 AM.png!
5. We suspect some credentials sync issue and manually synced the credentials and restarted the nodes with 0 requests, which fixed the problem.
Also, one few C* nodes we see below exception immediately after the bootstrap is completed and the process dies. is this contributing to the credentials issue?
NOTE: The C* nodes with zero traffic and the nodes with the below exception are not the same.
{quote}ERROR [main] 2019-12-12 05:34:40,412 CassandraDaemon.java:583 - Exception encountered during startup
java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: Undefined name salted_hash in selection clause
at org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:202) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.auth.Auth.setup(Auth.java:144) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:996) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391) [apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.16.jar:2.1.16]
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Undefined name salted_hash in selection clause
at org.apache.cassandra.cql3.statements.Selection.fromSelectors(Selection.java:292) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1592) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:198) ~[apache-cassandra-2.1.16.jar:2.1.16]
... 7 common frames omitted
{quote}
Not sure why this is happening, is this a potential bug or any other pointers to fix the problem.
C* Version: 2.1.16
Client: Datastax Java Driver.
system_auth RF: 3, dc-1:3 and dc-2:3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org