You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ross Black <ro...@gmail.com> on 2011/04/07 08:49:14 UTC

problem with large batch mutation set

Hi,

I am using the thrift client batch_mutate method with Cassandra 0.7.0  on
Ubuntu 10.10.

When the size of the mutations gets too large, the client fails with the
following exception:

Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
    at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
    at
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:157)
    at
org.apache.cassandra.thrift.Cassandra$Client.send_batch_mutate(Cassandra.java:901)
    at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:889)
    at
com.cxense.cxad.core.persistence.cassandra.store.BatchMutationTask.apply(BatchMutationTask.java:78)
    at
com.cxense.cxad.core.persistence.cassandra.store.BatchMutationTask.apply(BatchMutationTask.java:30)
    at
com.cxense.cassandra.conn.DefaultCassandraConnectionTemplate.execute(DefaultCassandraConnectionTemplate.java:316)
    at
com.cxense.cassandra.conn.DefaultCassandraConnectionTemplate.execute(DefaultCassandraConnectionTemplate.java:257)
    at
com.cxense.cxad.core.persistence.cassandra.store.AbstractCassandraStore.writeMutations(AbstractCassandraStore.java:492)
    ... 39 more
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
    at
org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
    ... 47 more


It took a while for me to discover that this obscure error message was as a
result of the thrift message exceeding the maximum frame size specified for
the Cassandra server. (default 15MB)
[Using TFastFramedTransport with a max frame size of 15728640 bytes.]

The poor error message looks like a bug in Thrift server code that assumes
that any transport exception is a connection failure which should drop the
connection.

*
*My main problem is how to ensure that this does not occur again in running
code.
I could configure the server with a larger frame size, but this size is
effectively arbitrary so there is no guarantee in our code would stop a very
large mutation being sent occasionally.
My current work-around is to break the mutation list into multiple parts,
but to do this correctly I need to track the size of each mutation which is
fairly messy.
*
Is there some way to configure Thrift or Cassandra to deal with messages
that are larger than the max frame size (at either client or server) ?*


Thanks,
Ross

Re: problem with large batch mutation set

Posted by Ryan King <ry...@twitter.com>.
On Wed, Apr 6, 2011 at 11:49 PM, Ross Black <ro...@gmail.com> wrote:

> Hi,
>
> I am using the thrift client batch_mutate method with Cassandra 0.7.0  on
> Ubuntu 10.10.
>
> When the size of the mutations gets too large, the client fails with the
> following exception:
>
> Caused by: org.apache.thrift.transport.TTransportException:
> java.net.SocketException: Connection reset
>     at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
>     at
> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:157)
>     at
> org.apache.cassandra.thrift.Cassandra$Client.send_batch_mutate(Cassandra.java:901)
>     at
> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:889)
>     at
> com.cxense.cxad.core.persistence.cassandra.store.BatchMutationTask.apply(BatchMutationTask.java:78)
>     at
> com.cxense.cxad.core.persistence.cassandra.store.BatchMutationTask.apply(BatchMutationTask.java:30)
>     at
> com.cxense.cassandra.conn.DefaultCassandraConnectionTemplate.execute(DefaultCassandraConnectionTemplate.java:316)
>     at
> com.cxense.cassandra.conn.DefaultCassandraConnectionTemplate.execute(DefaultCassandraConnectionTemplate.java:257)
>     at
> com.cxense.cxad.core.persistence.cassandra.store.AbstractCassandraStore.writeMutations(AbstractCassandraStore.java:492)
>     ... 39 more
> Caused by: java.net.SocketException: Connection reset
>     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
>     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>     at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>     at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
>     ... 47 more
>
>
> It took a while for me to discover that this obscure error message was as a
> result of the thrift message exceeding the maximum frame size specified for
> the Cassandra server. (default 15MB)
> [Using TFastFramedTransport with a max frame size of 15728640 bytes.]
>
> The poor error message looks like a bug in Thrift server code that assumes
> that any transport exception is a connection failure which should drop the
> connection.
>
> *
> *My main problem is how to ensure that this does not occur again in
> running code.
> I could configure the server with a larger frame size, but this size is
> effectively arbitrary so there is no guarantee in our code would stop a very
> large mutation being sent occasionally.
> My current work-around is to break the mutation list into multiple parts,
> but to do this correctly I need to track the size of each mutation which is
> fairly messy.
> *
> Is there some way to configure Thrift or Cassandra to deal with messages
> that are larger than the max frame size (at either client or server) ?*


The only way to do that is to set the frame size higher. Messages cannot be
bigger than the maximum frame size.

-ryan