[jira] [Commented] (CASSANDRA-14328) Invalid metadata has been detected for role

["Tania S Engel (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-14328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938924#comment-16938924 ] 

Tania S Engel commented on CASSANDRA-14328:
-------------------------------------------

I also had this issue happen on one of our nodes after it joined the Cassandra cluster. I confirmed by using another role that worked and connecting with cqlsh and running >use system_auth; select * from roles; 

I could see that in the roles can_login column the role had null (rather than true or false). This is the root cause of your exception.

In my case, joining looked as if it worked from our monitoring. nodetool netstats did show Mode: Joining but also showed "not sending any streams". nodetool netstats later was at Mode: normal.  {color:#172b4d}nodetool status showed it was an up node. {color}On closer inspection, the join had not streamed. I did have debug.log and it showed the node bootstrapped but it never logged "Creating new streaming plan for bootstrap.." so there was no streaming. Shortly after the bad node bootstrap started, the debug.log on the good existing seed node shows a socket closed and failure to connect. It did reconnect, but perhaps that is the reason the streaming plan never commenced. It is misleading that the bad node then continued to run and appear as if it was bootstrapped. 

I tried to fix the bad node by running >nodetool repair system_auth but that did not work. I was able to fix the roles with >nodetool --full system_auth. I was able to fix the remaining data by running a full repair on all tables.

> Invalid metadata has been detected for role
> -------------------------------------------
>
>                 Key: CASSANDRA-14328
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14328
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Legacy/CQL
>            Reporter: Pranav Jindal
>            Priority: Normal
>
> Cassandra Version : 3.10
> One node was replaced and was successfully up and working but CQL-SH fails with error.
>  
> CQL-SH error:
>  
> {code:java}
> Connection error: ('Unable to connect to any servers', {'10.180.0.150': AuthenticationFailed('Failed to authenticate to 10.180.0.150: Error from server: code=0000 [Server error] message="java.lang.RuntimeException: Invalid metadata has been detected for role utorjwcnruzzlzafxffgyqmlvkxiqcgb"',)})
> {code}
>  
> Cassandra server ERROR:
> {code:java}
> WARN [Native-Transport-Requests-1] 2018-03-20 13:37:17,894 CassandraRoleManager.java:96 - An invalid value has been detected in the roles table for role utorjwcnruzzlzafxffgyqmlvkxiqcgb. If you are unable to login, you may need to disable authentication and confirm that values in that table are accurate
> ERROR [Native-Transport-Requests-1] 2018-03-20 13:37:17,895 Message.java:623 - Unexpected exception during request; channel = [id: 0xdfc3604f, L:/10.180.0.150:9042 - R:/10.180.0.150:51668]
> java.lang.RuntimeException: Invalid metadata has been detected for role utorjwcnruzzlzafxffgyqmlvkxiqcgb
> at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:99) ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:82) ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:528) ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503) ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:310) ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.service.ClientState.login(ClientState.java:271) ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:80) ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.10.jar:3.10]
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.10.jar:3.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.NullPointerException: null
> at org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:273) ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:88) ~[apache-cassandra-3.10.jar:3.10]
> ... 16 common frames omitted
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org