You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jai Bheemsen Rao Dhanwada (Jira)" <ji...@apache.org> on 2019/12/12 19:19:00 UTC
[jira] [Updated] (CASSANDRA-15449) Credentials out of sync after
replacing the nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jai Bheemsen Rao Dhanwada updated CASSANDRA-15449:
--------------------------------------------------
Description:
Hello,
We are seeing a strange issue where, after replacing multiple C* nodes from the clusters intermittently we see an issue where few nodes doesn't have any credentials and the client queries fail.
Here are the sequence of steps
1. on a Multi DC C* cluster(12 nodes in each DC), we replaced all the nodes in one DC.
2. The approach we took to replace the nodes is kill one node and launch a new node with {{-Dcassandra.replace_address=}} and proceed with next node once the node is bootstrapped and CQL is enabled.
3. This process works fine and all of a sudden, we started seeing our application started failing with the below errors in the logs
{quote}com.datastax.driver.core.exceptions.UnauthorizedException: User abc has no SELECT permission on <table ks.cf> or any of its parents at com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:59) at com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:25) at
{quote}
4. At this stage we see that 3 nodes in the cluster takes zero traffic, while rest of the nodes are serving ~100 requests. (attached the metrics)
!Screen Shot 2019-12-12 at 11.13.52 AM.png!
5. We suspect some credentials sync issue and manually synced the credentials and restarted the nodes with 0 requests, which fixed the problem.
Also, one few C* nodes we see below exception immediately after the bootstrap is completed and the process dies. is this contributing to the credentials issue?
NOTE: The C* nodes with zero traffic and the nodes with the below exception are not the same.
{quote}ERROR [main] 2019-12-12 05:34:40,412 CassandraDaemon.java:583 - Exception encountered during startup
java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: Undefined name salted_hash in selection clause
at org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:202) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.auth.Auth.setup(Auth.java:144) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:996) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391) [apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.16.jar:2.1.16]
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Undefined name salted_hash in selection clause
at org.apache.cassandra.cql3.statements.Selection.fromSelectors(Selection.java:292) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1592) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:198) ~[apache-cassandra-2.1.16.jar:2.1.16]
... 7 common frames omitted
{quote}
Not sure why this is happening, is this a potential bug or any other pointers to fix the problem.
C* Version: 2.1.16
Client: Datastax Java Driver.
system_auth RF: 3, dc-1:3 and dc-2:3
was:
Hello,
We are seeing a strange issue where, after replacing multiple C* nodes from the clusters intermittently we see an issue where few nodes doesn't have any credentials and the client queries fail.
Here are the sequence of steps
1. on a Multi DC C* cluster(12 nodes in each DC), we replaced all the nodes in one DC.
2. The approach we took to replace the nodes is kill one node and launch a new node with {{-Dcassandra.replace_address=}} and proceed with next node once the node is bootstrapped and CQL is enabled.
3. This process works fine and all of a sudden, we started seeing our application started failing with the below errors in the logs
{quote}com.datastax.driver.core.exceptions.UnauthorizedException: User abc has no SELECT permission on <table ks.cf> or any of its parents at com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:59) at com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:25) at
{quote}
4. At this stage we see that 3 nodes in the cluster takes zero traffic, while rest of the nodes are serving ~100 requests. (attached the metrics)
!Screen Shot 2019-12-12 at 11.13.52 AM.png!
5. We suspect some credentials sync issue and manually synced the credentials and restarted the nodes with 0 requests, which fixed the problem.
Also, one few C* nodes we see below exception immediately after the bootstrap is completed and the process dies. is this contributing to the credentials issue?
NOTE: The C* nodes with zero traffic and the nodes with the below exception are not the same.
{quote}ERROR [main] 2019-12-12 05:34:40,412 CassandraDaemon.java:583 - Exception encountered during startup
java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: Undefined name salted_hash in selection clause
at org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:202) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.auth.Auth.setup(Auth.java:144) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:996) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391) [apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.16.jar:2.1.16]
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Undefined name salted_hash in selection clause
at org.apache.cassandra.cql3.statements.Selection.fromSelectors(Selection.java:292) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1592) ~[apache-cassandra-2.1.16.jar:2.1.16]
at org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:198) ~[apache-cassandra-2.1.16.jar:2.1.16]
... 7 common frames omitted
{quote}
Not sure why this is happening, is this a potential bug or any other pointers to fix the problem.
C* Version: 2.1.16
Client: Datastax Java Driver.
system_auth RF: 3, dc-1:3 and dc-2:3
> Credentials out of sync after replacing the nodes
> -------------------------------------------------
>
> Key: CASSANDRA-15449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15449
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jai Bheemsen Rao Dhanwada
> Priority: Normal
> Attachments: Screen Shot 2019-12-12 at 11.13.52 AM.png
>
>
> Hello,
> We are seeing a strange issue where, after replacing multiple C* nodes from the clusters intermittently we see an issue where few nodes doesn't have any credentials and the client queries fail.
> Here are the sequence of steps
> 1. on a Multi DC C* cluster(12 nodes in each DC), we replaced all the nodes in one DC.
> 2. The approach we took to replace the nodes is kill one node and launch a new node with {{-Dcassandra.replace_address=}} and proceed with next node once the node is bootstrapped and CQL is enabled.
> 3. This process works fine and all of a sudden, we started seeing our application started failing with the below errors in the logs
> {quote}com.datastax.driver.core.exceptions.UnauthorizedException: User abc has no SELECT permission on <table ks.cf> or any of its parents at com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:59) at com.datastax.driver.core.exceptions.UnauthorizedException.copy(UnauthorizedException.java:25) at
> {quote}
> 4. At this stage we see that 3 nodes in the cluster takes zero traffic, while rest of the nodes are serving ~100 requests. (attached the metrics)
> !Screen Shot 2019-12-12 at 11.13.52 AM.png!
> 5. We suspect some credentials sync issue and manually synced the credentials and restarted the nodes with 0 requests, which fixed the problem.
> Also, one few C* nodes we see below exception immediately after the bootstrap is completed and the process dies. is this contributing to the credentials issue?
> NOTE: The C* nodes with zero traffic and the nodes with the below exception are not the same.
> {quote}ERROR [main] 2019-12-12 05:34:40,412 CassandraDaemon.java:583 - Exception encountered during startup
> java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: Undefined name salted_hash in selection clause
> at org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:202) ~[apache-cassandra-2.1.16.jar:2.1.16]
> at org.apache.cassandra.auth.Auth.setup(Auth.java:144) ~[apache-cassandra-2.1.16.jar:2.1.16]
> at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:996) ~[apache-cassandra-2.1.16.jar:2.1.16]
> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.16.jar:2.1.16]
> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.16.jar:2.1.16]
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391) [apache-cassandra-2.1.16.jar:2.1.16]
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.16.jar:2.1.16]
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.16.jar:2.1.16]
> Caused by: org.apache.cassandra.exceptions.InvalidRequestException: Undefined name salted_hash in selection clause
> at org.apache.cassandra.cql3.statements.Selection.fromSelectors(Selection.java:292) ~[apache-cassandra-2.1.16.jar:2.1.16]
> at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:1592) ~[apache-cassandra-2.1.16.jar:2.1.16]
> at org.apache.cassandra.auth.PasswordAuthenticator.setup(PasswordAuthenticator.java:198) ~[apache-cassandra-2.1.16.jar:2.1.16]
> ... 7 common frames omitted
> {quote}
> Not sure why this is happening, is this a potential bug or any other pointers to fix the problem.
> C* Version: 2.1.16
> Client: Datastax Java Driver.
> system_auth RF: 3, dc-1:3 and dc-2:3
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org