[jira] [Commented] (CASSANDRA-15449) Credentials out of sync after replacing the nodes

["David Capwell (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997560#comment-16997560 ] 

David Capwell commented on CASSANDRA-15449:
-------------------------------------------

One thing to call out, it’s a good idea to run repair before replacing the host, that way all data is synced. 

You called out you manual synced the data, could you say how?  Was it repair or copy the files?

More detail would help figure out what happened, such as the logs. Without that I can only guess that repair didn’t happen before replace so data went missing on nodes. And the salted hash thing looks to be intermediate so curious if this is one time or happened a lot over a long period of time 

> Credentials out of sync after replacing the nodes
> -------------------------------------------------
>
>                 Key: CASSANDRA-15449
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-15449
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Jai Bheemsen Rao Dhanwada
>            Priority: Normal
>         Attachments: Screen Shot 2019-12-12 at 11.13.52 AM.png
>
>
> Hello,
> We are seeing a strange issue where, after replacing multiple C* nodes from the clusters intermittently we see an issue where few nodes doesn't have any credentials and the client queries fail.
> Here are the sequence of steps
> 1. on a Multi DC C* cluster(12 nodes in each DC), we replaced all the nodes in one DC. 
> 2. The approach we took to replace the nodes is kill one node and launch a new node with {{-Dcassandra.replace_address=}} and proceed with next node once the node is bootstrapped and CQL is enabled.
>  3. This process works fine and all of a sudden, we started seeing our application started failing with the below errors in the logs
> {quote}com.datastax.driver.core.exceptions.UnauthorizedException: User abc has no SELECT permission on <table ks.cf> or any of its parents at com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:59) at com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:25) at
> {quote}
> 4. At this stage we see that 3 nodes in the cluster takes zero traffic, while rest of the nodes are serving ~100 requests. (attached the metrics)
>  5. We suspect some credentials sync issue and manually synced the credentials and restarted the nodes with 0 requests, which fixed the problem.
> Also, one few C* nodes we see below exception immediately after the bootstrap is completed and the process dies. is this contributing to the credentials issue?
> NOTE:  The C* nodes with zero traffic and the nodes with the below exception are not the same.
> {quote}ERROR [main] 2019-12-12 05:34:40,412 CassandraDaemon.java:583 - Exception encountered during startup
>  java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: Undefined name salted_hash in selection clause
>  at org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:202) ~[apache-cassandra-2.1.16.jar:2.1.16]
>  at org.apache.cassandra.auth.Auth.setup(Auth.java:144) ~[apache-cassandra-2.1.16.jar:2.1.16]
>  at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:996) ~[apache-cassandra-2.1.16.jar:2.1.16]
>  at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.16.jar:2.1.16]
>  at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.16.jar:2.1.16]
>  at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391) [apache-cassandra-2.1.16.jar:2.1.16]
>  at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.16.jar:2.1.16]
>  at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.16.jar:2.1.16]
>  Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Undefined name salted_hash in selection clause
>  at org.apache.cassandra.cql3.statements.Selection.fromSelectors(Selection.java:292) ~[apache-cassandra-2.1.16.jar:2.1.16]
>  at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1592) ~[apache-cassandra-2.1.16.jar:2.1.16]
>  at org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:198) ~[apache-cassandra-2.1.16.jar:2.1.16]
>  ... 7 common frames omitted
> {quote}
> Not sure why this is happening, is this a potential bug or any other pointers to fix the problem.
> C* Version: 2.1.16
>  Client: Datastax Java Driver.
>  system_auth RF: 3, dc-1:3 and dc-2:3



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org