You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by 曹志富 <ca...@gmail.com> on 2015/10/16 17:30:55 UTC
Node down after some Exceptions
Hi,guys:
My C* cluster version is 2.1.6, vnodes, 25 nodes. JDK is 1.7.0_71, 4core,32
GB 1disk commitlog 1 disk sstable
In my cluster some nodes often down after some excpetions,and also cause
some others node down. The Exception is :
Unexpected exception during request
com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received o
nly 0 responses.
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
~[guava-16.0.jar:na]
at
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.authorize(ClientState.java:352)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:250)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:244)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:228)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:128)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.BatchStatement.checkAccess(BatchStatement.java:90)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:500)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.messages.BatchMessage.execute(BatchMessage.java:215)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
[apache-cassandra-2.1.6.jar:2.1.6]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_71]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.6.jar:2.1.6]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
~[guava-16.0.jar:na]
... 23 common frames omitted
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
at
org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:103)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:139)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1346)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1273)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1198)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:257)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265)
~[apache-cassandra-2.1.6.jar:2.1.6]
... 32 common frames omitted
WARN [GossipTasks:1] 2015-10-16 22:39:23,426 FailureDetector.java:251 -
Not marking nodes down due to local pause of 21768179618 > 5000000000
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
at
org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:103)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:139)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1346)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1273)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1198)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:257)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265)
~[apache-cassandra-2.1.6.jar:2.1.6]
... 13 common frames omitted
ERROR [SharedPool-Worker-16] 2015-10-16 22:39:23,123 Message.java:538 -
Unexpected exception during request; channel = [id: 0x5bd32250, /
172.19.141.2:44551 :> /172.19.105.51:904
2]
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.isExistingUser(Auth.java:86)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.login(ClientState.java:205)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:82)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
[apache-cassandra-2.1.6.jar:2.1.6]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_71]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.6.jar:2.1.6]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
at
org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:103)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:139)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1346)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1273)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1198)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:257)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265)
~[apache-cassandra-2.1.6.jar:2.1.6]
... 13 common frames omitted
ERROR [SharedPool-Worker-27] 2015-10-16 22:39:23,123 Message.java:538 -
Unexpected exception during request; channel = [id: 0x5ce52843, /
172.19.141.4:45114 :> /172.19.105.51:9042]
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.isExistingUser(Auth.java:86)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.login(ClientState.java:205)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:82)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
[apache-cassandra-2.1.6.jar:2.1.6]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_71]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.6.jar:2.1.6]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
at
org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:103)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:139)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1346)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1273)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1198)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:257)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265)
~[apache-cassandra-2.1.6.jar:2.1.6]
... 13 common frames omitted
ERROR [SharedPool-Worker-36] 2015-10-16 22:39:23,123 Message.java:538 -
Unexpected exception during request; channel = [id: 0xac59c3f8, /
172.19.141.6:41070 :> /172.19.105.51:9042]
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.isExistingUser(Auth.java:86)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.login(ClientState.java:205)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:82)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
[apache-cassandra-2.1.6.jar:2.1.6]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_71]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.6.jar:2.1.6]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
at
org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:103)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:139)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1346)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1273)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1198)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:257)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265)
~[apache-cassandra-2.1.6.jar:2.1.6]
... 13 common frames omitted
ERROR [SharedPool-Worker-28] 2015-10-16 22:39:23,425 ErrorMessage.java:251
- Unexpected exception during request
com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received o
nly 0 responses.
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
~[guava-16.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
~[guava-16.0.jar:na]
at
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:75)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.authorize(ClientState.java:352)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:250)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:244)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:228)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:128)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.BatchStatement.checkAccess(BatchStatement.java:90)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:500)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.messages.BatchMessage.execute(BatchMessage.java:215)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
[apache-cassandra-2.1.6.jar:2.1.6]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_71]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.6.jar:2.1.6]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_71]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.6.jar:2.1.6]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:276)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:97)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
~[guava-16.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
~[guava-16.0.jar:na]
... 23 common frames omitted
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
at
org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:103)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:139)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1346)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1273)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1198)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:257)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265)
~[apache-cassandra-2.1.6.jar:2.1.6]
... 32 common frames omitted
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1273)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1198)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:257)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
~[apache-cassandra-2.1.6.jar:2.1.6]
at org.apache.cassandra.auth.Auth.selectUser(Auth.java:265)
~[apache-cassandra-2.1.6.jar:2.1.6]
... 32 common frames omitted
The full log is system.log. config file is list in attach..
I have see an issue
https://issues.apache.org/jira/browse/CASSANDRA-7696.seems like My
situation.
Could Some One give me some advise to me.
--------------------------------------
Ranger Tsao