You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Per Otterström (JIRA)" <ji...@apache.org> on 2019/05/16 20:41:00 UTC
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if
authentication/authorization query fails
[ https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841705#comment-16841705 ]
Per Otterström commented on CASSANDRA-15041:
--------------------------------------------
So, this is mostly a cosmetic issue in the sense that there is no way to resolve the underlying problem - that not enough replicas are available to read from the system_auth tables. Still, this generates error messages that cause operators to jump, and Cassandra isn't behaving well towards clients when this happens.
I've updated the auth_test.py to use consistent cache settings and created a few new test cases to reproduce this issue in the scenario where not enough replicas are available. They revile unwanted behavior with small variation on different releases. In short:
* TC1 will trigger Cassandra to perform a background update of a cached credentials/roles/permissions entries.
* TC2 will trigger authorization when cached entries have passed both update-interval and validity (blocking update)
* TC3 will trigger authorization when cache is disabled.
Link to dtest [patch|https://github.com/apache/cassandra-dtest/compare/master...eperott:cassandra-15041].
I'm expecting Cassandra to fail gracefully on TC1, possibly with a warning, but no stack trace.
I'm expecting TC2 and TC3 to reject the request with exception to indicate not-authorized|unavailable|timeout, not sure which makes most sense. In any case, TC2 and TC3 should fail the same way and there should be no errors or stack traces in the log.
Results on different releases:
* 4.0: TC2 fail
* 3.11: TC1, TC2, TC3 and existing test_login fail
* 3.0: TC1, TC2 and TC3 fail
* 2.2: TC1, TC2 and TC3 fail
4.0 behaves generally better since a similar ticket as this one was fixed in CASSANDRA-13113. The reason TC2 fails is that the request actually will be authorized, even though the cached entries should have timed out. From what I can tell the cache is handing out stale entries.
The reason test_login fail on 3.11 branch is that we're caching credentials since 3.4.
All new test cases fail on 3.11, 3.0 and 2.2 as reported in this ticket.
So far I've made no attempt to work on a fix for this. Before we dive into that, I'd like to get some feedback on the dtests and my findings above.
TC2 and TC3 currently expect an UnavailableException and a message similar to "Cannot achieve consistency level QUORUM" at the client side, simply because this is the behavior in 4.0 branch. I feel this might confuse users a bit as the exception and message is related to the internal lookup on the system_auth.* table, rather than the actual query sent to the cluster. Would it make more sense to throw back an UnauthorizedException?
[~beobal] and [~ifesdjeen], you were both much involved in CASSANDRA-13113. What are your thoughts on this?
> UncheckedExecutionException if authentication/authorization query fails
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
> Reporter: Per Otterström
> Priority: Normal
>
> If cache update for permissions/credentials/roles fails with UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537 ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4]
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:172) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
> ... 26 common frames omitted
> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:518) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:283) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:172) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
> ... 40 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1766) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1728) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:526) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:508) ~[apache-cassandra-3.11.4.jar:3.11.4]
> ... 47 common frames omitted
> {noformat}
> Also, if {{x_validity_in_ms}} > {{x_update_interval_in_ms}}, then the background update thread will fail in a similar way:
> {noformat}
> ERROR [PermissionsCacheRefresh:1] 2019-03-04 16:30:43,541 CassandraDaemon.java:228 - Exception in thread Thread[PermissionsCacheRefresh:1,5,main]
> java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:518) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.isSuper(CassandraRoleManager.java:307) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:52) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthCache$1.lambda$reload$0(AuthCache.java:180) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1766) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1728) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:526) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:508) ~[apache-cassandra-3.11.4.jar:3.11.4]
> ... 11 common frames omitted
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org