Hector Could not flush transport error

["Senthil, Athinanthny X. -ND" <At...@disney.com>]

We get the below error in app logs when it's trying to hit DC which doesn't get traffic in multi DC cluster. But other DC where all traffic hits seems to be responding without any issues. App read/write requests are timing out. Hector client version is 1.0.5. On cassandra side , we are getting socket timeouts. Configs are all same in both DC and on app side as well?

We expanded this cluster by adding 3 nodes to 2 DC . It has 6 nodes in each DC.  App did not recognize new nodes until it was explicitly defined in property file. Since then we are seeing the issue/error below. This DC acts like failover DC and this is only place where get socket timeout errors in cassandra layer in all nodes.

Do we need to change any config ? Will increasing this value thrift_framed_transport_size_in_mb: 15 help?

Tried enabling assandra.useSocketKeepalive=true, timeout still occurs. cassandraThriftSocketTimeout=0 in app property


2014-02-06 10:38:57,936 -0800 [Hector.me.prettyprint.cassandra.connection.NodeAutoDiscoverService-1] ERROR me.prettyprint.cassandra.connection.client.HThriftClient - Could not flush transport (to be expected if the pool is shutting
down) in close for client: CassandraClient<:9160-10220>
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:156)
at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:98)
at me.prettyprint.cassandra.connection.client.HThriftClient.close(HThriftClient.java:26)
at me.prettyprint.cassandra.connection.HConnectionManager.closeClient(HConnectionManager.java:323)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:272)
at me.prettyprint.cassandra.service.AbstractCluster.describeKeyspaces(AbstractCluster.java:136)
at me.prettyprint.cassandra.connection.NodeAutoDiscoverService.discoverNodes(NodeAutoDiscoverService.java:97)
at me.prettyprint.cassandra.connection.NodeAutoDiscoverService.doAddNodes(NodeAutoDiscoverService.java:68)
at me.prettyprint.cassandra.connection.NodeAutoDiscoverService$QueryRing.run(NodeAutoDiscoverService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)

RE: Hector Could not flush transport error

["Senthil, Athinanthny X. -ND" <At...@disney.com>]

Version is C 1.2.6. We use DSE 3.1.3

From: Robert Coli [mailto:rcoli@eventbrite.com]
Sent: Friday, February 07, 2014 4:17 PM
To: user@cassandra.apache.org
Subject: Re: Hector Could not flush transport error

On Fri, Feb 7, 2014 at 4:05 PM, Senthil, Athinanthny X. -ND <At...@disney.com>> wrote:
We get the below error in app logs when it's trying to hit DC which doesn't get traffic in multi DC cluster.

What version of Cassandra?

=Rob

Re: Hector Could not flush transport error

[Robert Coli <rc...@eventbrite.com>]

On Fri, Feb 7, 2014 at 4:05 PM, Senthil, Athinanthny X. -ND <
Athinanthny.X.Senthil.-ND@disney.com> wrote:

> We get the below error in app logs when it's trying to hit DC which
> doesn't get traffic in multi DC cluster.
>

What version of Cassandra?

=Rob