You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Philippe <wa...@gmail.com> on 2011/08/07 23:06:29 UTC
batch mutates & throughput
A question regarding batch mutates and how others might be throttling the
system to prevent timeouts.
My 3-node, RF=3 cluster has been performing ok while bulk loading data
(applying counter updates). I've been able to run 16 threads in parallel
that each perform about 400 mutates/s on a loaded cluster.
Then I thought, hey, let's get rid of the network round trip and batch this
thing...
So I converted my code to use a mutator and addCounter instead of
insertCounter (on Hector). However, when I do, the results are always bad.
When I execute()
- every 5000 lines, I get wonderful performance but I constantly get
Timeouts
- every 500, same thing
- every 10, the timeouts take longer to appear but they're still there
- every 1, it works just like before batching
And this happens even with a single thread running
So my question is not about the absolute performance of my cluster but about
how I'm supposed to use batch updates : it doesn't look like the execute()
call blocks until it's performed the mutation and tpstats has showed up to
200.000 mutations pending.
Any ideas ?
Re: batch mutates & throughput
Posted by Philippe <wa...@gmail.com>.
>
> Hmm, can you create a ticket with a simple way to reproduce that? We
> should be giving back an InvalidRequestException for
> multiple-mutations-on-same-key instead of erroring out later and
> causing timeouts.
>
Humm... this is actually quite confusing. When I look at the error, I don't
see the same super column although it is the same columns. It does look like
it's the same key. I thought that was possible.
Isn't this really
https://issues.apache.org/jira/browse/CASSANDRA-2949except I get a
sligthly different log message ?
Sylvain, is this really the same ? Any idea of when 8.1.3 will be voted on ?
ERROR [ReplicateOnWriteStage:409] 2011-08-08 16:49:11,182
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[ReplicateOnWriteStage:409,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: ColumnFamily
ColumnFamily(PUBLIC_MONTHLY_17 [SuperColumn(ghcisco
[00000000010000:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 4,
8}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 1,
2}]@1312814951133!-9223372036854775808,00000000010001:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd,
4, -776}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 1,
-194}]@1312814951133!-9223372036854775808,]),]) already has modifications in
this mutation: ColumnFamily(PUBLIC_MONTHLY_17 [SuperColumn(gdwls
[00000000010000:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 4,
8}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 1,
2}]@1312814951133!-9223372036854775808,00000000010001:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd,
4, -756}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 1,
-189}]@1312814951133!-9223372036854775808,]),])
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException: ColumnFamily
ColumnFamily(PUBLIC_MONTHLY_17 [SuperColumn(ghcisco
[00000000010000:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 4,
8}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 1,
2}]@1312814951133!-9223372036854775808,00000000010001:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd,
4, -776}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 1,
-194}]@1312814951133!-9223372036854775808,]),]) already has modifications in
this mutation: ColumnFamily(PUBLIC_MONTHLY_17 [SuperColumn(gdwls
[00000000010000:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd, 4,
8}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 1,
2}]@1312814951133!-9223372036854775808,00000000010001:false:[{223a4de0-b5fb-11e0-0000-826f85850cbd,
4, -756}*,{224ceb80-b5fb-11e0-0000-848783ceb9bf, 1,
-189}]@1312814951133!-9223372036854775808,]),])
at org.apache.cassandra.db.RowMutation.add(RowMutation.java:123)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:120)
at
org.apache.cassandra.service.StorageProxy$5$1.runMayThrow(StorageProxy.java:455)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
>
> On Mon, Aug 8, 2011 at 12:34 AM, Philippe <wa...@gmail.com> wrote:
> > Hi Boris,
> > Thanks for the suggestion, I didn't know there was one.
> > I believe have finally figured it out and it turns out my last two
> questions
> > are related.
> > First, my batch loading was ignoring a bunch of rows when reading the
> first
> > file (so it took hundreds of potential mutations for the problem to show
> up)
> > and secondly, the ReplicateOnWriteStage error was generated by the batch
> > mutations themselves and explained the TimedOutException : I was doing
> > multiple mutations on the same key in one batch
> >
> >
> > 2011/8/8 Boris Yen <yu...@gmail.com>
> >>
> >> Maybe you could try to adjust the setting "cassandraThriftSocketTimeout"
> >> of hector. https://github.com/rantav/hector/wiki/User-Guide
> >>
> >> On Mon, Aug 8, 2011 at 6:54 AM, Philippe <wa...@gmail.com> wrote:
> >>>
> >>> Quick followup.
> >>> I have pushed the RPC timeout to 30s. Using Hector, I'm doing 1 thread
> >>> doing batches of 10 mutates at a time so that's even slower than when I
> was
> >>> doing 16 threads in parallel doing non-batched mutations.
> >>> After a couple hundred execute() calls, I get a timeout for every node;
> I
> >>> have a 15 second grace period between retries. tpstats indicate no
> pendings
> >>> on any of the nodes. I never recover from that
> >>> I then set the batch size to one and it seems to work a lot better. The
> >>> only difference I note is that the Mutator.execute() method returns a
> result
> >>> than sometimes has a null host and 0 microsecond time in the batch
> sizes of
> >>> ten but never in batch sizes of 1.
> >>>
> >>> I'm stumped ! Any ideas ?
> >>> Thanks
> >>
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: batch mutates & throughput
Posted by Philippe <wa...@gmail.com>.
>
> Hmm, can you create a ticket with a simple way to reproduce that? We
> should be giving back an InvalidRequestException for
> multiple-mutations-on-same-key instead of erroring out later and
> causing timeouts.
>
Will do but I'm supposed to be on vacation for a couple weeks now so it will
take a while to sanitize stuff before posting it.
This is now hitting me in another part of the app where I had batched
stuff... oh well
On Mon, Aug 8, 2011 at 12:34 AM, Philippe <wa...@gmail.com> wrote:
> > Hi Boris,
> > Thanks for the suggestion, I didn't know there was one.
> > I believe have finally figured it out and it turns out my last two
> questions
> > are related.
> > First, my batch loading was ignoring a bunch of rows when reading the
> first
> > file (so it took hundreds of potential mutations for the problem to show
> up)
> > and secondly, the ReplicateOnWriteStage error was generated by the batch
> > mutations themselves and explained the TimedOutException : I was doing
> > multiple mutations on the same key in one batch
> >
> >
> > 2011/8/8 Boris Yen <yu...@gmail.com>
> >>
> >> Maybe you could try to adjust the setting "cassandraThriftSocketTimeout"
> >> of hector. https://github.com/rantav/hector/wiki/User-Guide
> >>
> >> On Mon, Aug 8, 2011 at 6:54 AM, Philippe <wa...@gmail.com> wrote:
> >>>
> >>> Quick followup.
> >>> I have pushed the RPC timeout to 30s. Using Hector, I'm doing 1 thread
> >>> doing batches of 10 mutates at a time so that's even slower than when I
> was
> >>> doing 16 threads in parallel doing non-batched mutations.
> >>> After a couple hundred execute() calls, I get a timeout for every node;
> I
> >>> have a 15 second grace period between retries. tpstats indicate no
> pendings
> >>> on any of the nodes. I never recover from that
> >>> I then set the batch size to one and it seems to work a lot better. The
> >>> only difference I note is that the Mutator.execute() method returns a
> result
> >>> than sometimes has a null host and 0 microsecond time in the batch
> sizes of
> >>> ten but never in batch sizes of 1.
> >>>
> >>> I'm stumped ! Any ideas ?
> >>> Thanks
> >>
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: batch mutates & throughput
Posted by Jonathan Ellis <jb...@gmail.com>.
Hmm, can you create a ticket with a simple way to reproduce that? We
should be giving back an InvalidRequestException for
multiple-mutations-on-same-key instead of erroring out later and
causing timeouts.
On Mon, Aug 8, 2011 at 12:34 AM, Philippe <wa...@gmail.com> wrote:
> Hi Boris,
> Thanks for the suggestion, I didn't know there was one.
> I believe have finally figured it out and it turns out my last two questions
> are related.
> First, my batch loading was ignoring a bunch of rows when reading the first
> file (so it took hundreds of potential mutations for the problem to show up)
> and secondly, the ReplicateOnWriteStage error was generated by the batch
> mutations themselves and explained the TimedOutException : I was doing
> multiple mutations on the same key in one batch
>
>
> 2011/8/8 Boris Yen <yu...@gmail.com>
>>
>> Maybe you could try to adjust the setting "cassandraThriftSocketTimeout"
>> of hector. https://github.com/rantav/hector/wiki/User-Guide
>>
>> On Mon, Aug 8, 2011 at 6:54 AM, Philippe <wa...@gmail.com> wrote:
>>>
>>> Quick followup.
>>> I have pushed the RPC timeout to 30s. Using Hector, I'm doing 1 thread
>>> doing batches of 10 mutates at a time so that's even slower than when I was
>>> doing 16 threads in parallel doing non-batched mutations.
>>> After a couple hundred execute() calls, I get a timeout for every node; I
>>> have a 15 second grace period between retries. tpstats indicate no pendings
>>> on any of the nodes. I never recover from that
>>> I then set the batch size to one and it seems to work a lot better. The
>>> only difference I note is that the Mutator.execute() method returns a result
>>> than sometimes has a null host and 0 microsecond time in the batch sizes of
>>> ten but never in batch sizes of 1.
>>>
>>> I'm stumped ! Any ideas ?
>>> Thanks
>>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: batch mutates & throughput
Posted by Philippe <wa...@gmail.com>.
Hi Boris,
Thanks for the suggestion, I didn't know there was one.
I believe have finally figured it out and it turns out my last two questions
are related.
First, my batch loading was ignoring a bunch of rows when reading the first
file (so it took hundreds of potential mutations for the problem to show up)
and secondly, the ReplicateOnWriteStage error was generated by the batch
mutations themselves and explained the TimedOutException : I was doing
multiple mutations on the same key in one batch
2011/8/8 Boris Yen <yu...@gmail.com>
> Maybe you could try to adjust the setting "cassandraThriftSocketTimeout"
> of hector. https://github.com/rantav/hector/wiki/User-Guide
>
>
> On Mon, Aug 8, 2011 at 6:54 AM, Philippe <wa...@gmail.com> wrote:
>
>> Quick followup.
>> I have pushed the RPC timeout to 30s. Using Hector, I'm doing 1 thread
>> doing batches of 10 mutates at a time so that's even slower than when I was
>> doing 16 threads in parallel doing non-batched mutations.
>> After a couple hundred execute() calls, I get a timeout for every node; I
>> have a 15 second grace period between retries. tpstats indicate no pendings
>> on any of the nodes. I never recover from that
>>
>> I then set the batch size to one and it seems to work a lot better. The
>> only difference I note is that the Mutator.execute() method returns a result
>> than sometimes has a null host and 0 microsecond time in the batch sizes of
>> ten but never in batch sizes of 1.
>>
>>
>> I'm stumped ! Any ideas ?
>>
>> Thanks
>>
>
>
Re: batch mutates & throughput
Posted by Boris Yen <yu...@gmail.com>.
Maybe you could try to adjust the setting "cassandraThriftSocketTimeout" of
hector. https://github.com/rantav/hector/wiki/User-Guide
On Mon, Aug 8, 2011 at 6:54 AM, Philippe <wa...@gmail.com> wrote:
> Quick followup.
> I have pushed the RPC timeout to 30s. Using Hector, I'm doing 1 thread
> doing batches of 10 mutates at a time so that's even slower than when I was
> doing 16 threads in parallel doing non-batched mutations.
> After a couple hundred execute() calls, I get a timeout for every node; I
> have a 15 second grace period between retries. tpstats indicate no pendings
> on any of the nodes. I never recover from that
>
> I then set the batch size to one and it seems to work a lot better. The
> only difference I note is that the Mutator.execute() method returns a result
> than sometimes has a null host and 0 microsecond time in the batch sizes of
> ten but never in batch sizes of 1.
>
>
> I'm stumped ! Any ideas ?
>
> Thanks
>
Re: batch mutates & throughput
Posted by Philippe <wa...@gmail.com>.
Quick followup.
I have pushed the RPC timeout to 30s. Using Hector, I'm doing 1 thread doing
batches of 10 mutates at a time so that's even slower than when I was doing
16 threads in parallel doing non-batched mutations.
After a couple hundred execute() calls, I get a timeout for every node; I
have a 15 second grace period between retries. tpstats indicate no pendings
on any of the nodes. I never recover from that
I then set the batch size to one and it seems to work a lot better. The only
difference I note is that the Mutator.execute() method returns a result than
sometimes has a null host and 0 microsecond time in the batch sizes of ten
but never in batch sizes of 1.
I'm stumped ! Any ideas ?
Thanks