Broken pipe when variating a lot number of connections

[Rodrigo Felix <ro...@gmail.com>]

Hi,

    I've been changing a benchmarking tool (YCSB) to vary the number of
clients throughout a workload execution and, for some reason, I believe
Cassandra is facing some problems to handle the variation (both up and
down) on the number of connections. Each client has a connection and
clients are added or removed during execution.
    For some reason, after some (variable) time, I get the following
exception, using the original Cassandra client implemented by YCSB, that
works for a constant number of clients.
    Is there any reason or known issue to explain why cassandra does not
handle properly the number of connection and gives broken pipe? I'm
supposing this can be a Cassandra problem, but feel free to let me know if
you think I'm wrong.
   Thanks in advance.

   *Cassandra version:* 1.1.5
   *Some property values:*

   - rpc_keepalive: true
   - rpc_min_threads: 16
   - rpc_max_threads: 2048

    *Exception:*

*Changing clients from 192 to 240*
*org.apache.thrift.transport.TTransportException: java.net.SocketException:
Broken pipe*
*org.apache.thrift.transport.TTransportException: java.net.SocketException:
Broken pipe*
* at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
*
* at
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
*
* at
org.apache.cassandra.thrift.Cassandra$Client.send_set_keyspace(Cassandra.java:436)
*
* at
org.apache.cassandra.thrift.Cassandra$Client.set_keyspace(Cassandra.java:425)
*
* at com.yahoo.ycsb.db.CassandraClient10.scan(CassandraClient10.java:314)*
* at com.yahoo.ycsb.DBWrapper.scan(DBWrapper.java:106)*
* at
com.yahoo.ycsb.workloads.CoreWorkload.doTransactionScan(CoreWorkload.java:530)
*
* at
com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:431)*
* at com.yahoo.ycsb.ClientThread.run(ClientThread.java:105)*
Att.

*Rodrigo Felix de Almeida*
LSBD - Universidade Federal do Ceará
Project Manager
MBA, CSM, CSPO, SCJP

Re: Broken pipe when variating a lot number of connections

[Edward Capriolo <ed...@gmail.com>]

The YCSB client is not very advanced. Hector, asynax or the native driver
will work better. There are a few ycsb forks as each nosql person usually
needs to fork ycsb to get the most out of it, check github.


On Thu, Apr 11, 2013 at 6:28 PM, Rodrigo Felix <
rodrigofelixdealmeida@gmail.com> wrote:

> Hi,
>
>     I've been changing a benchmarking tool (YCSB) to vary the number of
> clients throughout a workload execution and, for some reason, I believe
> Cassandra is facing some problems to handle the variation (both up and
> down) on the number of connections. Each client has a connection and
> clients are added or removed during execution.
>     For some reason, after some (variable) time, I get the following
> exception, using the original Cassandra client implemented by YCSB, that
> works for a constant number of clients.
>     Is there any reason or known issue to explain why cassandra does not
> handle properly the number of connection and gives broken pipe? I'm
> supposing this can be a Cassandra problem, but feel free to let me know if
> you think I'm wrong.
>    Thanks in advance.
>
>    *Cassandra version:* 1.1.5
>    *Some property values:*
>
>    - rpc_keepalive: true
>    - rpc_min_threads: 16
>    - rpc_max_threads: 2048
>
>     *Exception:*
>
> *Changing clients from 192 to 240*
> *org.apache.thrift.transport.TTransportException:
> java.net.SocketException: Broken pipe*
> *org.apache.thrift.transport.TTransportException:
> java.net.SocketException: Broken pipe*
> * at
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
> *
> * at
> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158)
> *
> * at
> org.apache.cassandra.thrift.Cassandra$Client.send_set_keyspace(Cassandra.java:436)
> *
> * at
> org.apache.cassandra.thrift.Cassandra$Client.set_keyspace(Cassandra.java:425)
> *
> * at com.yahoo.ycsb.db.CassandraClient10.scan(CassandraClient10.java:314)*
> * at com.yahoo.ycsb.DBWrapper.scan(DBWrapper.java:106)*
> * at
> com.yahoo.ycsb.workloads.CoreWorkload.doTransactionScan(CoreWorkload.java:530)
> *
> * at
> com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:431)
> *
> * at com.yahoo.ycsb.ClientThread.run(ClientThread.java:105)*
> Att.
>
> *Rodrigo Felix de Almeida*
> LSBD - Universidade Federal do Ceará
> Project Manager
> MBA, CSM, CSPO, SCJP
>