You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Viktor Jevdokimov <Vi...@adform.com> on 2012/02/01 09:48:41 UTC
Consurrent compactors
Hi,
When concurrent compactors are set to more then 1, it's rare when more than 1 compaction is running in parallel.
Didn't checked the source code, but it looks like when next compaction task (any of minor, major, or cleanup) is for the same CF, it will not start in parallel and next tasks are not checked.
Will it be possible to check all tasks, not only the next one, to find which of them can be started?
This is actual especially when nightly cleanup is running, a lot of cleanup tasks are pending, regular minor compactions are waiting until all cleanup compactions are finished.
Best regards/ Pagarbiai
Viktor Jevdokimov
Senior Developer
Email: Viktor.Jevdokimov@adform.com
Phone: +370 5 212 3063. Fax: +370 5 261 0453
J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
[Adform news]<http://www.adform.com/>
[Visit us!]
Follow:
[twitter]<http://twitter.com/#!/adforminsider>
Visit our blog<http://www.adform.com/site/blog>
Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Re: Consurrent compactors
Posted by aaron morton <aa...@thelastpickle.com>.
Not sure I understand the question. Do you have an example where a CF is not getting compacted ?
The compaction tasks will be processed in the order they are submitted. If you have concurrent_compactors > 1 then the thread pool for compactions (excluding validation compactions) will be able to process multiple compaction tasks in parallel.
If you have a CF that gets a lot more traffic than other CF's it will require more compaction. But by running concurrent_compactors > 1 smaller CF's should still be able to get through.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 3/02/2012, at 9:15 PM, Viktor Jevdokimov wrote:
> My concern is not anout cleanup, but about supposed „tendency of small sstables to accumulate during a single long running compactions“. When next task is for the same column family as currently long-running compaction, other column families compactions are freezed and concurrent_compactors > 1setting just not working.
>
>
> Best regards/ Pagarbiai
>
> Viktor Jevdokimov
> Senior Developer
>
> Email: Viktor.Jevdokimov@adform.com
> Phone: +370 5 212 3063. Fax: +370 5 261 0453
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
>
>
> <signature-logo29.png>
>
> <dm-exco4823.png>
> Follow:
>
> <tweet18be.png>
> Visit our blog
>
> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Wednesday, February 01, 2012 21:51
> To: user@cassandra.apache.org
> Subject: Re: Consurrent compactors
>
> (Assuming 1.0* release)
> From the comments in cassandra.yaml
>
> # Number of simultaneous compactions to allow, NOT including
> # validation "compactions" for anti-entropy repair. Simultaneous
> # compactions can help preserve read performance in a mixed read/write
> # workload, by mitigating the tendency of small sstables to accumulate
> # during a single long running compactions. The default is usually
> # fine and if you experience problems with compaction running too
> # slowly or too fast, you should look at
> # compaction_throughput_mb_per_sec first.
> #
> # This setting has no effect on LeveledCompactionStrategy.
> #
> # concurrent_compactors defaults to the number of cores.
> # Uncomment to make compaction mono-threaded, the pre-0.8 default.
> #concurrent_compactors: 1
>
> If you set it to 1 then only 1 compaction should run at a time, excluding validation.
>
> How often do you run a cleanup compaction ? They are only necessary when you perform a token move.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 1/02/2012, at 9:48 PM, Viktor Jevdokimov wrote:
>
>
> Hi,
>
> When concurrent compactors are set to more then 1, it’s rare when more than 1 compaction is running in parallel.
>
> Didn’t checked the source code, but it looks like when next compaction task (any of minor, major, or cleanup) is for the same CF, it will not start in parallel and next tasks are not checked.
>
> Will it be possible to check all tasks, not only the next one, to find which of them can be started?
>
> This is actual especially when nightly cleanup is running, a lot of cleanup tasks are pending, regular minor compactions are waiting until all cleanup compactions are finished.
>
>
>
> Best regards/ Pagarbiai
>
> Viktor Jevdokimov
> Senior Developer
>
> Email: Viktor.Jevdokimov@adform.com
> Phone: +370 5 212 3063. Fax: +370 5 261 0453
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
>
>
> <signature-logo29.png>
>
> <dm-exco4823.png>
> Follow:
>
> <tweet18be.png>
> Visit our blog
> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
>
>
RE: Consurrent compactors
Posted by Viktor Jevdokimov <Vi...@adform.com>.
My concern is not anout cleanup, but about supposed "tendency of small sstables to accumulate during a single long running compactions". When next task is for the same column family as currently long-running compaction, other column families compactions are freezed and concurrent_compactors > 1 setting just not working.
Best regards/ Pagarbiai
Viktor Jevdokimov
Senior Developer
Email: Viktor.Jevdokimov@adform.com
Phone: +370 5 212 3063. Fax: +370 5 261 0453
J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
[Adform news]<http://www.adform.com/>
[Visit us!]
Follow:
[twitter]<http://twitter.com/#!/adforminsider>
Visit our blog<http://www.adform.com/site/blog>
Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, February 01, 2012 21:51
To: user@cassandra.apache.org
Subject: Re: Consurrent compactors
(Assuming 1.0* release)
>From the comments in cassandra.yaml
# Number of simultaneous compactions to allow, NOT including
# validation "compactions" for anti-entropy repair. Simultaneous
# compactions can help preserve read performance in a mixed read/write
# workload, by mitigating the tendency of small sstables to accumulate
# during a single long running compactions. The default is usually
# fine and if you experience problems with compaction running too
# slowly or too fast, you should look at
# compaction_throughput_mb_per_sec first.
#
# This setting has no effect on LeveledCompactionStrategy.
#
# concurrent_compactors defaults to the number of cores.
# Uncomment to make compaction mono-threaded, the pre-0.8 default.
#concurrent_compactors: 1
If you set it to 1 then only 1 compaction should run at a time, excluding validation.
How often do you run a cleanup compaction ? They are only necessary when you perform a token move.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 1/02/2012, at 9:48 PM, Viktor Jevdokimov wrote:
Hi,
When concurrent compactors are set to more then 1, it's rare when more than 1 compaction is running in parallel.
Didn't checked the source code, but it looks like when next compaction task (any of minor, major, or cleanup) is for the same CF, it will not start in parallel and next tasks are not checked.
Will it be possible to check all tasks, not only the next one, to find which of them can be started?
This is actual especially when nightly cleanup is running, a lot of cleanup tasks are pending, regular minor compactions are waiting until all cleanup compactions are finished.
Best regards/ Pagarbiai
Viktor Jevdokimov
Senior Developer
Email: Viktor.Jevdokimov@adform.com<ma...@adform.com>
Phone: +370 5 212 3063. Fax: +370 5 261 0453
J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
<signature-logo29.png><http://www.adform.com/>
<dm-exco4823.png>
Follow:
<tweet18be.png><http://twitter.com/#!/adforminsider>
Visit our blog<http://www.adform.com/site/blog>
Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Re: Consurrent compactors
Posted by aaron morton <aa...@thelastpickle.com>.
(Assuming 1.0* release)
From the comments in cassandra.yaml
# Number of simultaneous compactions to allow, NOT including
# validation "compactions" for anti-entropy repair. Simultaneous
# compactions can help preserve read performance in a mixed read/write
# workload, by mitigating the tendency of small sstables to accumulate
# during a single long running compactions. The default is usually
# fine and if you experience problems with compaction running too
# slowly or too fast, you should look at
# compaction_throughput_mb_per_sec first.
#
# This setting has no effect on LeveledCompactionStrategy.
#
# concurrent_compactors defaults to the number of cores.
# Uncomment to make compaction mono-threaded, the pre-0.8 default.
#concurrent_compactors: 1
If you set it to 1 then only 1 compaction should run at a time, excluding validation.
How often do you run a cleanup compaction ? They are only necessary when you perform a token move.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 1/02/2012, at 9:48 PM, Viktor Jevdokimov wrote:
> Hi,
>
> When concurrent compactors are set to more then 1, it’s rare when more than 1 compaction is running in parallel.
>
> Didn’t checked the source code, but it looks like when next compaction task (any of minor, major, or cleanup) is for the same CF, it will not start in parallel and next tasks are not checked.
>
> Will it be possible to check all tasks, not only the next one, to find which of them can be started?
>
> This is actual especially when nightly cleanup is running, a lot of cleanup tasks are pending, regular minor compactions are waiting until all cleanup compactions are finished.
>
>
>
> Best regards/ Pagarbiai
>
> Viktor Jevdokimov
> Senior Developer
>
> Email: Viktor.Jevdokimov@adform.com
> Phone: +370 5 212 3063. Fax: +370 5 261 0453
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
>
>
> <signature-logo29.png>
>
> <dm-exco4823.png>
> Follow:
>
> <tweet18be.png>
> Visit our blog
>
> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
>
>