error sending update to orkash1:9997

["mohit.kaushik" <mo...@orkash.com>]

Dear All,
/
//error sending update to orkash1:9997: 
org.apache.thrift.transport.TTransportException: 
java.net.SocketTimeoutException: 120000 millis timeout while waiting for 
channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/192.168.10.122:59662 
remote=orkash1/192.168.10.121:9997]/

I found the following error in the monitor logs for all three servers 
today morning out of which one was dead (orkash3)  that may be caused by 
network issues. which I am trying to diagnose and shows the message 
before it dies

GC pause checker not called in a timely fashion. Expected every 30.0 seconds but was 160.6 seconds since last check

unable to get tablet server status orkash3:9997[1523a7ae07e0081] org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection timed out

Lost tablet server lock (reason = SESSION_EXPIRED), exiting.

But other two servers are not having any network issues. shows error 
sending updates and 36000 mutations rejected yesterday.

One more thing I am not getting is whenever (everytime) I compact a 
table it says

  WARN : Thread "shell" stuck on IO to orkash5:9999 (0) for at least 
120084 ms

and after a long ( 4 - 5 hours) it completes with msg

INFO : Thread "shell" no longer stuck on IO to orkash5:9999 (0) sawError 
= false

I do not have any idea why shell always stuck for big tables and why 
there are errors on sending updates to other server. Please through some 
light..

there are some more errors/warn that might help/relate.

Tracing spans are being dropped because there are already 5000 spans queued for delivery.
This does not affect performance, security or data integrity, but distributed tracing information is being lost.

Thread "gc" stuck on IO to orkash5:9999 (0) for at least 120429 ms //on orkashh2

2016-01-22 16:33:57,825 [gc.SimpleGarbageCollector] INFO : List of 
delete candidates has exceeded the memory threshold. Attempting to 
delete what has been gathered so far.
Should I need to make some GC configurations?? The ingest rate is not so 
high for now.


Thanks
Mohit Kaushik