You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Per Otterström (JIRA)" <ji...@apache.org> on 2019/06/10 20:25:00 UTC
[jira] [Commented] (CASSANDRA-15041) UncheckedExecutionException if
authentication/authorization query fails
[ https://issues.apache.org/jira/browse/CASSANDRA-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16860306#comment-16860306 ]
Per Otterström commented on CASSANDRA-15041:
--------------------------------------------
Had to make some adjustments while implementing this.
When we fail to perform authorization it is not always possible to convert whatever-exception-we-get into an {{UnavailableException}} since the {{UnavailableException}} constructor requires a bunch of parameters (CL plus required and live nodes). I didn't feel comfortable to change this to achieve our goals here, so I went with the other proposal to convert this into an {{UnauthorizedException}} instead. But I'm happy to discuss options. Worth considering, since IAuthorizer is a public plug-in interface it should define a generic behavior. And, for example, it would be somewhat awkward for an {{LDAPAuthorizer}} to throw an {{UnavailableException}} if it fails to contact the LDAP server, so the {{UnauthorizedException}} may be a better fit anyway.
A side effect of signaling {{UnauthorizedException}} instead of {{UnavailableException}} is that the issue with the stale entries from the Caffeine cache don'ẗ show any more. This is because the driver will not retry on {{UnauthrizedException}}, and the Caffeine issue only shows if it get repeated queries on failing keys. But IMO we should still see to this. I created CASSANDRA-15153 for this.
Also, had a setback with one of the goals of this ticket - to make the background cache reload thread mute if it fails. Turns out the error message is buried deep down in the Guava {{LoadingCache}}. Only option I see for pre-4.0 is to mute this in the logback config.
PR for [dtest|https://github.com/apache/cassandra-dtest/pull/52].
The patches for Cassandra differs a bit on 2.2/3.0 vs. 3.11 vs. trunk. Not sure what's the best way to provide these patches to simplify review and merge into upstream repo. Below are links to the individual branches on my github clone without merge commits, is that OK? Lots if dtests are timing out since I only have the free service, but will try to run failing tests locally
||Patch||CI||
|[15041-cassandra-2.2|https://github.com/eperott/cassandra/tree/15041-cassandra-2.2]|[CircleCI|https://circleci.com/gh/eperott/workflows/cassandra/tree/cci%2F15041-cassandra-2.2]|
|[15041-cassandra-3.0|https://github.com/eperott/cassandra/tree/15041-cassandra-3.0]|[CircleCI|https://circleci.com/gh/eperott/workflows/cassandra/tree/cci%2F15041-cassandra-3.0]|
|[15041-cassandra-3.11|https://github.com/eperott/cassandra/tree/15041-cassandra-3.11]|[CircleCI|https://circleci.com/gh/eperott/workflows/cassandra/tree/cci%2F15041-cassandra-3.11]|
|[15041-trunk|https://github.com/eperott/cassandra/tree/15041-trunk]|[CircleCI|https://circleci.com/gh/eperott/workflows/cassandra/tree/cci%2F15041-trunk]|
> UncheckedExecutionException if authentication/authorization query fails
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-15041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15041
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
> Reporter: Per Otterström
> Assignee: Per Otterström
> Priority: Normal
> Fix For: 2.2.15, 3.0.19, 3.11.5, 4.0
>
>
> If cache update for permissions/credentials/roles fails with UnavailableException this comes back to client as UncheckedExecutionException.
> Stack trace on server side:
> {noformat}
> ERROR [Native-Transport-Requests-1] 2019-03-04 16:30:51,537 ErrorMessage.java:384 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:222) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:532) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:509) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:146) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4]
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:97) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:172) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
> ... 26 common frames omitted
> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:518) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:283) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:172) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
> ... 40 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1766) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1728) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:526) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:508) ~[apache-cassandra-3.11.4.jar:3.11.4]
> ... 47 common frames omitted
> {noformat}
> Also, if {{x_validity_in_ms}} > {{x_update_interval_in_ms}}, then the background update thread will fail in a similar way:
> {noformat}
> ERROR [PermissionsCacheRefresh:1] 2019-03-04 16:30:43,541 CassandraDaemon.java:228 - Exception in thread Thread[PermissionsCacheRefresh:1,5,main]
> java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:518) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.isSuper(CassandraRoleManager.java:307) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:52) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.AuthCache$1.lambda$reload$0(AuthCache.java:180) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.4.jar:3.11.4]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
> at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:334) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:162) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.<init>(StorageProxy.java:1766) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1728) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1671) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1586) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1209) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:315) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:285) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:526) ~[apache-cassandra-3.11.4.jar:3.11.4]
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:508) ~[apache-cassandra-3.11.4.jar:3.11.4]
> ... 11 common frames omitted
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org