You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Shannon Carey (JIRA)" <ji...@apache.org> on 2019/04/16 22:10:00 UTC
[jira] [Commented] (CASSANDRA-12735)
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out
[ https://issues.apache.org/jira/browse/CASSANDRA-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819544#comment-16819544 ]
Shannon Carey commented on CASSANDRA-12735:
-------------------------------------------
Adjusting the cache validity period and refresh rate only helps so much. A configuration should be exposed to allow control over the timeout duration for this. If the timeout is too small, then it repeatedly tries reading the data, making it even less likely to succeed.
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out
> -------------------------------------------------------------------------
>
> Key: CASSANDRA-12735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12735
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core, Local/Config
> Environment: Python 2.7.11, Datastax Cassandra 3.7.0
> Reporter: Rajesh Radhakrishnan
> Priority: Normal
> Fix For: 3.11.x
>
>
> We got a cluster of two nodes running Cassandra.3.7.0 and using client running Python 2.7.11 injecting lot of data from maybe 100 or so jobs.
> ------------------------------------------------------------------------------------------
> Cache setting can be seen from system.log:
> INFO [main] 2016-09-30 15:12:50,002 AuthCache.java:172 - (Re)initializing CredentialsCache (validity period/update interval/max entries) (2000/2000/1000)
> INFO [SharedPool-Worker-1] 2016-09-30 15:15:09,561 AuthCache.java:172 - (Re)initializing PermissionsCache (validity period/update interval/max entries) (10000/10000/1000)
> INFO [SharedPool-Worker-1] 2016-09-30 15:15:24,319 AuthCache.java:172 - (Re)initializing RolesCache (validity period/update interval/max entries) (5000/5000/1000)
> ===========================================================
> But I am getting the following exception :
> ERROR [SharedPool-Worker-90] 2016-09-30 15:17:20,883 ErrorMessage.java:338 - Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.ClientState.authorize(ClientState.java:375) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:308) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:285) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:272) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:256) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.cql3.statements.BatchStatement.checkAccess(BatchStatement.java:137) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:502) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:495) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.transport.messages.BatchMessage.execute(BatchMessage.java:217) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [apache-cassandra-3.7.0.jar:3.7.0]
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292) [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32) [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:283) [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_73]
> at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.7.0.jar:3.7.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:86) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.PermissionsCache.lambda$new$1(PermissionsCache.java:37) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
> ... 26 common frames omitted
> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:489) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:269) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.RolesCache.lambda$new$195(RolesCache.java:36) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
> ... 40 common frames omitted
> Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
> at org.apache.cassandra.service.ReadCallback.awaitResults(ReadCallback.java:132) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:137) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1714) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1663) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1604) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1523) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:979) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:273) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:234) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:497) ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:485) ~[apache-cassandra-3.7.0.jar:3.7.0]
> ... 47 common frames omitted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org