socket timeout errors in one DC in a multi DC cluster

["Senthil, Athinanthny X. -ND" <At...@disney.com>]

Experiencing socket timeout errors in one  DC in most of the nodes in multi dc cluster. Here is error. Client is having intermittent high response time issues in this DC.     DC1 does not experience any timeout issues, but DC2 does though.  This error started occurring recently and repeats consecutive days. Any suggestions on cause for it?

During this situation, when we try to run queries on CQL3 on local server itself, sometime we get rpc_timeout errors. But this is intermittent as well.


ERROR [Thrift:97]   CustomTThreadPoolServer.java (line 219) Error occurred during processing of message.
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 2 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
at org.apache.cassandra.service.ClientState.authorize(ClientState.java:308)
at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:178)
at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:171)
at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:155)
at org.apache.cassandra.thrift.CassandraServer.createMutationList(CassandraServer.java:681)
at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:749)
at com.datastax.bdp.server.DseServer.batch_mutate(DseServer.java:931)
at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3622)
at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3610)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)