You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Henrik Schröder <sk...@gmail.com> on 2013/02/19 14:56:52 UTC
Testing compaction strategies on a single production server?
Hey,
Version 1.1 of Cassandra introduced live traffic sampling, which allows you
to measure the performance of a node without it really joining the cluster:
http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-1-live-traffic-sampling
That page mentions that you can change the compaction strategy through jmx
if you want to test out a different strategy on your survey node.
That's great, but it doesn't give you a complete view of how your
performance would change, since you're not doing reads from the survey
node. But what would happen if you used jmx to change the compaction
strategy of a column family on a single *production* node? Would that be a
safe way to test it out or are there side-effects of doing that live?
And if you do that, would running a major compaction transform the entire
column family to the new format?
Finally, if the test was a success, how do you proceed from there? Just
change the schema?
/Henrik
Re: Testing compaction strategies on a single production server?
Posted by Henrik Schröder <sk...@gmail.com>.
Thanks Aaron,
I hear you on the unchartered territory bit, we're definitely not gonna
risk our live data unless we know it's safe to do what we suggested. :-) Oh
well, I guess we'll have to setup a survey node instead.
/Henrik
On Thu, Feb 21, 2013 at 4:54 AM, aaron morton <aa...@thelastpickle.com>wrote:
> I *think* it will work. The steps in the blog post to change the
> compaction strategy before RING_DELAY expires is to ensure no sstables are
> created before the strategy is changed.
>
> But I think you will be venturing into unchartered territory where their
> might be dragons. And not the fun Disney kind.
>
> While it may be more work I personally would use one node in write survey
> to test LCS
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/02/2013, at 6:28 AM, Henrik Schröder <sk...@gmail.com> wrote:
>
> Well, that answer didn't really help. I know how to make a survey node,
> and I know how to simulate reads to it, it's just that that's a lot of
> work, and I wouldn't be sure that the simulated load is the same as the
> production load.
>
> We gather a lot of metrics from our production servers, so we know exactly
> how they perform over long periods of time. Changing a single server to run
> a different compaction strategy would allow us to know in detail how a
> different strategy would impact the cluster.
>
> So, is it possible to modify org.apache.cassandra.db.[keyspace].[column
> family].CompactionStrategyClass through jmx on a production server without
> any ill effects? Or is this only possible to do on a survey node while it
> is in a specific state?
>
>
> /Henrik
>
>
> On Tue, Feb 19, 2013 at 3:09 PM, Viktor Jevdokimov <
> Viktor.Jevdokimov@adform.com> wrote:
>
>> Just turn off dynamic snitch on survey node and make read requests from
>> it directly with CL.ONE, watch histograms, compare.****
>>
>> ** **
>>
>> Regarding switching compaction strategy there’re a lot of info already.**
>> **
>>
>> ** **
>>
>> ** **
>> Best regards / Pagarbiai
>> *Viktor Jevdokimov*
>> Senior Developer
>>
>> Email: Viktor.Jevdokimov@adform.com
>> Phone: +370 5 212 3063, Fax +370 5 261 0453
>> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
>> Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider>
>> Take a ride with Adform's Rich Media Suite<http://vimeo.com/adform/richmedia>
>> <signature-logo18be.png> <http://www.adform.com/>
>> <signature-best-employer-logo6784.png>
>> <http://www.adform.com/site/blog/adform/adform-takes-top-spot-in-best-employer-survey/>
>>
>> Disclaimer: The information contained in this message and attachments is
>> intended solely for the attention and use of the named addressee and may be
>> confidential. If you are not the intended recipient, you are reminded that
>> the information remains the property of the sender. You must not use,
>> disclose, distribute, copy, print or rely on this e-mail. If you have
>> received this message in error, please contact the sender immediately and
>> irrevocably delete this message and any copies.
>>
>> *From:* Henrik Schröder [mailto:skrolle@gmail.com]
>> *Sent:* Tuesday, February 19, 2013 15:57
>> *To:* user
>> *Subject:* Testing compaction strategies on a single production server?**
>> **
>>
>> ** **
>>
>> Hey,****
>>
>>
>> Version 1.1 of Cassandra introduced live traffic sampling, which allows
>> you to measure the performance of a node without it really joining the
>> cluster:
>> http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-1-live-traffic-sampling
>> ****
>>
>> That page mentions that you can change the compaction strategy through
>> jmx if you want to test out a different strategy on your survey node.
>>
>> That's great, but it doesn't give you a complete view of how your
>> performance would change, since you're not doing reads from the survey
>> node. But what would happen if you used jmx to change the compaction
>> strategy of a column family on a single *production* node? Would that be a
>> safe way to test it out or are there side-effects of doing that live?
>>
>> And if you do that, would running a major compaction transform the entire
>> column family to the new format?****
>>
>> Finally, if the test was a success, how do you proceed from there? Just
>> change the schema?
>>
>> ****
>>
>> /Henrik****
>>
>
>
>
Re: Testing compaction strategies on a single production server?
Posted by aaron morton <aa...@thelastpickle.com>.
I *think* it will work. The steps in the blog post to change the compaction strategy before RING_DELAY expires is to ensure no sstables are created before the strategy is changed.
But I think you will be venturing into unchartered territory where their might be dragons. And not the fun Disney kind.
While it may be more work I personally would use one node in write survey to test LCS
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 20/02/2013, at 6:28 AM, Henrik Schröder <sk...@gmail.com> wrote:
> Well, that answer didn't really help. I know how to make a survey node, and I know how to simulate reads to it, it's just that that's a lot of work, and I wouldn't be sure that the simulated load is the same as the production load.
>
> We gather a lot of metrics from our production servers, so we know exactly how they perform over long periods of time. Changing a single server to run a different compaction strategy would allow us to know in detail how a different strategy would impact the cluster.
>
> So, is it possible to modify org.apache.cassandra.db.[keyspace].[column family].CompactionStrategyClass through jmx on a production server without any ill effects? Or is this only possible to do on a survey node while it is in a specific state?
>
>
> /Henrik
>
>
> On Tue, Feb 19, 2013 at 3:09 PM, Viktor Jevdokimov <Vi...@adform.com> wrote:
> Just turn off dynamic snitch on survey node and make read requests from it directly with CL.ONE, watch histograms, compare.
>
>
>
> Regarding switching compaction strategy there’re a lot of info already.
>
>
>
>
>
> Best regards / Pagarbiai
> Viktor Jevdokimov
> Senior Developer
>
> Email: Viktor.Jevdokimov@adform.com
> Phone: +370 5 212 3063, Fax +370 5 261 0453
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
> Follow us on Twitter: @adforminsider
> Take a ride with Adform's Rich Media Suite
> <signature-logo18be.png>
> <signature-best-employer-logo6784.png>
>
> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
>
> From: Henrik Schröder [mailto:skrolle@gmail.com]
> Sent: Tuesday, February 19, 2013 15:57
> To: user
> Subject: Testing compaction strategies on a single production server?
>
>
>
> Hey,
>
>
> Version 1.1 of Cassandra introduced live traffic sampling, which allows you to measure the performance of a node without it really joining the cluster: http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-1-live-traffic-sampling
>
> That page mentions that you can change the compaction strategy through jmx if you want to test out a different strategy on your survey node.
>
> That's great, but it doesn't give you a complete view of how your performance would change, since you're not doing reads from the survey node. But what would happen if you used jmx to change the compaction strategy of a column family on a single *production* node? Would that be a safe way to test it out or are there side-effects of doing that live?
>
> And if you do that, would running a major compaction transform the entire column family to the new format?
>
> Finally, if the test was a success, how do you proceed from there? Just change the schema?
>
>
> /Henrik
>
>
Re: Testing compaction strategies on a single production server?
Posted by Henrik Schröder <sk...@gmail.com>.
Well, that answer didn't really help. I know how to make a survey node, and
I know how to simulate reads to it, it's just that that's a lot of work,
and I wouldn't be sure that the simulated load is the same as the
production load.
We gather a lot of metrics from our production servers, so we know exactly
how they perform over long periods of time. Changing a single server to run
a different compaction strategy would allow us to know in detail how a
different strategy would impact the cluster.
So, is it possible to modify org.apache.cassandra.db.[keyspace].[column
family].CompactionStrategyClass through jmx on a production server without
any ill effects? Or is this only possible to do on a survey node while it
is in a specific state?
/Henrik
On Tue, Feb 19, 2013 at 3:09 PM, Viktor Jevdokimov <
Viktor.Jevdokimov@adform.com> wrote:
> Just turn off dynamic snitch on survey node and make read requests from
> it directly with CL.ONE, watch histograms, compare.****
>
> ** **
>
> Regarding switching compaction strategy there’re a lot of info already.***
> *
>
> ** **
>
> ** **
> Best regards / Pagarbiai
> *Viktor Jevdokimov*
> Senior Developer
>
> Email: Viktor.Jevdokimov@adform.com
> Phone: +370 5 212 3063, Fax +370 5 261 0453
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
> Follow us on Twitter: @adforminsider <http://twitter.com/#!/adforminsider>
> Take a ride with Adform's Rich Media Suite<http://vimeo.com/adform/richmedia>
> [image: Adform News] <http://www.adform.com>
> [image: Adform awarded the Best Employer 2012]
> <http://www.adform.com/site/blog/adform/adform-takes-top-spot-in-best-employer-survey/>
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
> *From:* Henrik Schröder [mailto:skrolle@gmail.com]
> *Sent:* Tuesday, February 19, 2013 15:57
> *To:* user
> *Subject:* Testing compaction strategies on a single production server?***
> *
>
> ** **
>
> Hey,****
>
>
> Version 1.1 of Cassandra introduced live traffic sampling, which allows
> you to measure the performance of a node without it really joining the
> cluster:
> http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-1-live-traffic-sampling
> ****
>
> That page mentions that you can change the compaction strategy through jmx
> if you want to test out a different strategy on your survey node.
>
> That's great, but it doesn't give you a complete view of how your
> performance would change, since you're not doing reads from the survey
> node. But what would happen if you used jmx to change the compaction
> strategy of a column family on a single *production* node? Would that be a
> safe way to test it out or are there side-effects of doing that live?
>
> And if you do that, would running a major compaction transform the entire
> column family to the new format?****
>
> Finally, if the test was a success, how do you proceed from there? Just
> change the schema?
>
> ****
>
> /Henrik****
>
RE: Testing compaction strategies on a single production server?
Posted by Viktor Jevdokimov <Vi...@adform.com>.
Just turn off dynamic snitch on survey node and make read requests from it directly with CL.ONE, watch histograms, compare.
Regarding switching compaction strategy there're a lot of info already.
Best regards / Pagarbiai
Viktor Jevdokimov
Senior Developer
Email: Viktor.Jevdokimov@adform.com<ma...@adform.com>
Phone: +370 5 212 3063, Fax +370 5 261 0453
J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
Follow us on Twitter: @adforminsider<http://twitter.com/#!/adforminsider>
Take a ride with Adform's Rich Media Suite<http://vimeo.com/adform/richmedia>
[Adform News] <http://www.adform.com>
[Adform awarded the Best Employer 2012] <http://www.adform.com/site/blog/adform/adform-takes-top-spot-in-best-employer-survey/>
Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
From: Henrik Schröder [mailto:skrolle@gmail.com]
Sent: Tuesday, February 19, 2013 15:57
To: user
Subject: Testing compaction strategies on a single production server?
Hey,
Version 1.1 of Cassandra introduced live traffic sampling, which allows you to measure the performance of a node without it really joining the cluster: http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-1-live-traffic-sampling
That page mentions that you can change the compaction strategy through jmx if you want to test out a different strategy on your survey node.
That's great, but it doesn't give you a complete view of how your performance would change, since you're not doing reads from the survey node. But what would happen if you used jmx to change the compaction strategy of a column family on a single *production* node? Would that be a safe way to test it out or are there side-effects of doing that live?
And if you do that, would running a major compaction transform the entire column family to the new format?
Finally, if the test was a success, how do you proceed from there? Just change the schema?
/Henrik