You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Roshan Dawrani <ro...@gmail.com> on 2011/03/10 15:30:34 UTC
mutator.execute() timings - big variance noted - pointers needed on
understanding/improving it
Hi,
I am in the middle of some load testing on a 1-node Cassandra setup. We are
not on very high loads yet. We have recorded the timings taken up by
mutator.execute() calls and we see this kind of variation during the test
run:
So, 25% of the times, execute() calls come back in 25 milli-seconds, but the
longer calls go upto 4 seconds.
Can someone please provide some pointers on what and where to focus on in my
Hector / Cassandra setup? We are mostly on the default Cassandra
configuration at this time - only change is the max connection pool size
(CassandraHostConfigurator.maxActive) is changed to 300 from a default of
50.
I would also like to add that the time increase is not linear - it starts
fast, goes, slow, very slow, and becomes faster again.
------------------------
25% 29
50% 105
66% 185
70% 208
75% 240
80% 297
90% 510
95% 854
98% 1075
99% 1215
100% 4442
------------------------
--
Roshan
Blog: http://roshandawrani.wordpress.com/
Twitter: @roshandawrani <http://twitter.com/roshandawrani>
Skype: roshandawrani
Re: mutator.execute() timings - big variance noted - pointers needed
on understanding/improving it
Posted by Roshan Dawrani <ro...@gmail.com>.
Hi All,
Thanks for the inputs. I will start investigating this morning with the help
of these.
Regards,
Roshan
On Fri, Mar 11, 2011 at 2:49 AM, aaron morton <aa...@thelastpickle.com>wrote:
> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
>
> <http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts>Aaron
>
> On 11 Mar 2011, at 05:08, sridhar basam wrote:
>
>
> Sounds like GC from your description of fast->slow->fast. Collect GC times
> from both the client and server side and plot against your application
> timing.
>
> If you uncomment the verbose GC entries in the cassandra-env.sh file you
> should get timing for the server side, pass in the same arguments for your
> client. Align time across the 3 files and plot to see if GC is the cause.
>
> Sridhar
>
>
>
> On Thu, Mar 10, 2011 at 9:30 AM, Roshan Dawrani <ro...@gmail.com>wrote:
>
>> Hi,
>>
>> I am in the middle of some load testing on a 1-node Cassandra setup. We
>> are not on very high loads yet. We have recorded the timings taken up by
>> mutator.execute() calls and we see this kind of variation during the test
>> run:
>>
>> So, 25% of the times, execute() calls come back in 25 milli-seconds, but
>> the longer calls go upto 4 seconds.
>>
>> Can someone please provide some pointers on what and where to focus on in
>> my Hector / Cassandra setup? We are mostly on the default Cassandra
>> configuration at this time - only change is the max connection pool size
>> (CassandraHostConfigurator.maxActive) is changed to 300 from a default of
>> 50.
>>
>> I would also like to add that the time increase is not linear - it starts
>> fast, goes, slow, very slow, and becomes faster again.
>>
>> ------------------------
>> 25% 29
>> 50% 105
>> 66% 185
>> 70% 208
>> 75% 240
>> 80% 297
>> 90% 510
>> 95% 854
>> 98% 1075
>> 99% 1215
>> 100% 4442
>> ------------------------
>>
>> --
>> Roshan
>> Blog: http://roshandawrani.wordpress.com/
>> Twitter: @roshandawrani <http://twitter.com/roshandawrani>
>> Skype: roshandawrani
>>
>>
>
>
Re: mutator.execute() timings - big variance noted - pointers needed on understanding/improving it
Posted by aaron morton <aa...@thelastpickle.com>.
http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
Aaron
On 11 Mar 2011, at 05:08, sridhar basam wrote:
>
> Sounds like GC from your description of fast->slow->fast. Collect GC times from both the client and server side and plot against your application timing.
>
> If you uncomment the verbose GC entries in the cassandra-env.sh file you should get timing for the server side, pass in the same arguments for your client. Align time across the 3 files and plot to see if GC is the cause.
>
> Sridhar
>
>
>
> On Thu, Mar 10, 2011 at 9:30 AM, Roshan Dawrani <ro...@gmail.com> wrote:
> Hi,
>
> I am in the middle of some load testing on a 1-node Cassandra setup. We are not on very high loads yet. We have recorded the timings taken up by mutator.execute() calls and we see this kind of variation during the test run:
>
> So, 25% of the times, execute() calls come back in 25 milli-seconds, but the longer calls go upto 4 seconds.
>
> Can someone please provide some pointers on what and where to focus on in my Hector / Cassandra setup? We are mostly on the default Cassandra configuration at this time - only change is the max connection pool size (CassandraHostConfigurator.maxActive) is changed to 300 from a default of 50.
>
> I would also like to add that the time increase is not linear - it starts fast, goes, slow, very slow, and becomes faster again.
>
> ------------------------
> 25% 29
> 50% 105
> 66% 185
> 70% 208
> 75% 240
> 80% 297
> 90% 510
> 95% 854
> 98% 1075
> 99% 1215
> 100% 4442
> ------------------------
>
> --
> Roshan
> Blog: http://roshandawrani.wordpress.com/
> Twitter: @roshandawrani
> Skype: roshandawrani
>
>
Re: mutator.execute() timings - big variance noted - pointers needed
on understanding/improving it
Posted by sridhar basam <sr...@basam.org>.
Sounds like GC from your description of fast->slow->fast. Collect GC times
from both the client and server side and plot against your application
timing.
If you uncomment the verbose GC entries in the cassandra-env.sh file you
should get timing for the server side, pass in the same arguments for your
client. Align time across the 3 files and plot to see if GC is the cause.
Sridhar
On Thu, Mar 10, 2011 at 9:30 AM, Roshan Dawrani <ro...@gmail.com>wrote:
> Hi,
>
> I am in the middle of some load testing on a 1-node Cassandra setup. We are
> not on very high loads yet. We have recorded the timings taken up by
> mutator.execute() calls and we see this kind of variation during the test
> run:
>
> So, 25% of the times, execute() calls come back in 25 milli-seconds, but
> the longer calls go upto 4 seconds.
>
> Can someone please provide some pointers on what and where to focus on in
> my Hector / Cassandra setup? We are mostly on the default Cassandra
> configuration at this time - only change is the max connection pool size
> (CassandraHostConfigurator.maxActive) is changed to 300 from a default of
> 50.
>
> I would also like to add that the time increase is not linear - it starts
> fast, goes, slow, very slow, and becomes faster again.
>
> ------------------------
> 25% 29
> 50% 105
> 66% 185
> 70% 208
> 75% 240
> 80% 297
> 90% 510
> 95% 854
> 98% 1075
> 99% 1215
> 100% 4442
> ------------------------
>
> --
> Roshan
> Blog: http://roshandawrani.wordpress.com/
> Twitter: @roshandawrani <http://twitter.com/roshandawrani>
> Skype: roshandawrani
>
>