[jira] [Commented] (CASSANDRA-11469) dtest failure in upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test

["Sam Tunnicliffe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307580#comment-15307580 ] 

Sam Tunnicliffe commented on CASSANDRA-11469:
---------------------------------------------

These failures seem to be concurrency related, in particular there are races in the way that schema mutations are applied. What I believe is happening here is this: {{PasswordAuthenticator}} uses the presence of {{system_auth.credentials}} to indicate that the legacy credentials storage is still in place, as the data conversion performed on upgrade to 2.2 has not yet completed. Once all nodes are upgraded, it's the responsibility of an operator to drop the legacy tables, at which point {{PasswordAuthenticator}} switches to read the hashed password from {{system_auth.roles}} table. The race condition we seem to be hitting in these cases is that the {{credentials}} table is removed from the local schema datastructures after {{PasswordAuthenticator}} [tests for its existence|https://github.com/apache/cassandra/blob/cassandra-2.2.6/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L82], but before the actual [read is performed|https://github.com/apache/cassandra/blob/cassandra-2.2.6/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L143].

A simple solution is to have {{PasswordAuthenticator}} branch on the existence of {{system_auth.users}} rather than {{credentials}}, as both are migrated together and once a node is on 2.2+, it will never read the {{users}} table, as all user info is then read from {{roles}}. 

I've made that trivial change in [here|https://github.com/beobal/cassandra/commit/2445864c0ef8fc2a75d7ba10c6e6e76e0efad219] and [modified the dtest|https://github.com/beobal/cassandra-dtest/commit/4fa0813cfba3391f20045115d76e1cdf7379c0a0] so that it uses that as the upgrade target. Re-running the [multiplexed job|http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/125/parameters/] using that dtest branch shows no occurrences of this particular failure. There are a small number of other failures, which I can investigate though.

[~philipthompson], at first glance, it looks as though the failures are actually being doubly reported. Am I reading that correctly, that there are only 3 failures, not 6 as reported? 

> dtest failure in upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_22_test
> ------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-11469
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11469
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Philip Thompson
>            Assignee: Sam Tunnicliffe
>              Labels: dtest
>             Fix For: 2.2.x
>
>
> Upgrading from 2.1 to 2.2, we are seeing failures in upgrade_internal_auth_test.py. I have seen a variety of stack traces, but the most common is:
> {code}
> java.lang.IllegalArgumentException: Unknown keyspace/cf pair (system_auth.credentials)
> 	at org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:169) ~[main/:na]
> 	at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1383) ~[main/:na]
> 	at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1277) ~[main/:na]
> 	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:221) ~[main/:na]
> 	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:176) ~[main/:na]
> 	at org.apache.cassandra.auth.PasswordAuthenticator.doAuthenticate(PasswordAuthenticator.java:143) ~[main/:na]
> 	at org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:85) ~[main/:na]
> 	at org.apache.cassandra.auth.PasswordAuthenticator.access$100(PasswordAuthenticator.java:53) ~[main/:na]
> 	at org.apache.cassandra.auth.PasswordAuthenticator$PlainTextSaslAuthenticator.getAuthenticatedUser(PasswordAuthenticator.java:181) ~[main/:na]
> 	at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:78) ~[main/:na]
> 	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na]
> 	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na]
> 	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51]
> 	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [main/:na]
> 	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> {code}
> This failure is flaky, but not uncommon. I ran it 300 times here, and saw about 100 failures on CI:
> http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/53/testReport/
> This is a recent failure, and given the type of failure, it seems like potentially a bug and not a test issue. [~beobal], you may be most interested in looking at this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)