You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Flavien Charlon <fl...@gmail.com> on 2015/01/18 20:51:06 UTC
Compaction failing to trigger
Hi,
I am using Size Tier Compaction (Cassandra 2.1.2). Minor compaction is not
triggering even though it should. See the SSTables on disk:
http://pastebin.com/PSwZ5mrT
You can see that we have 41 SSTable between 60MB and 85MB, which should
trigger compaction unless I am missing something.
Is that a bug?
Thanks,
Flavien
Re: Compaction failing to trigger
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Jan 21, 2015 at 10:10 AM, Flavien Charlon <flavien.charlon@gmail.com
> wrote:
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>>
>>
>
> This doesn't really answer my question, I asked whether this particular
> bug (which I can't find in JIRA) is planned to be fixed in 2.1.3, not
> whether 2.1.3 would be production ready.
>
No idea, but as I didn't recognize your name/email and you were
encountering problems with an IMO not-ready-for-production version. Many
people who are new to Cassandra and pre-or-close-to-production might be
better served by running a slightly older version and focusing on the
challenge of writing their app against a mostly-working distributed
database instead of troubleshooting Cassandra bugs.
tl;dr - Cassandra bugs in cutting edge versions are likely best encountered
by experienced operators who can recognize them and respond, not new
operators.
While we're on this topic, the version numbering is very misleading.
> Version which are not recommended for production should be very explicitly
> labelled as such (beta for example), and 2.1.0 should really be what you
> call now 2.1.6.
>
That's why I wrote the blog post. It is however important to note that I
speak in no official capacity for Apache Cassandra or Datastax.
The intent of the project is for x.y.0 to be production ready, and in
fairness they have recently added new QA processes which are likely to
drive the production ready version down from x.y.6. They are only human,
however, and as human developers are likely have slightly different (lower)
standards for production readiness than the typical operator. I wrote that
blog post to help set operator-appropriate expectations, so people are not
disappointed with the overall stability of Cassandra.
I personally operate Cassandra slightly on the trailing edge, and as a
result only encounter a limited subset of the problems I assist people with
on the list and IRC.
=Rob
Re: Compaction failing to trigger
Posted by Flavien Charlon <fl...@gmail.com>.
>
> What version of Cassandra are you running?
2.1.2
Are they all "live"? Are there pending compactions, or exceptions regarding
> compactions in your logs?
Yes they are all live according to cfstats. There is no pending compaction
or exception in the logs.
https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
This doesn't really answer my question, I asked whether this particular bug
(which I can't find in JIRA) is planned to be fixed in 2.1.3, not whether
2.1.3 would be production ready.
While we're on this topic, the version numbering is very misleading.
Version which are not recommended for production should be very explicitly
labelled as such (beta for example), and 2.1.0 should really be what you
call now 2.1.6.
Setting 'cold_reads_to_omit' to 0 did the job for me
Thanks, I've tried it, and it works. This should probably be made the
default IMO.
Flavien
On 20 January 2015 at 22:51, Eric Stevens <mi...@gmail.com> wrote:
> @Rob - he's probably referring to the thread titled "Reasons for nodes not
> compacting?" where Tyler speculates that the tables are falling below the
> cold read threshold for compaction. He speculated it may be a bug. At the
> same time in a different thread, Roland had a similar problem, and Tyler's
> proposed workaround seemed to work for him.
>
> On Tue, Jan 20, 2015 at 3:35 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Sun, Jan 18, 2015 at 6:06 PM, Flavien Charlon <
>> flavien.charlon@gmail.com> wrote:
>>
>>> It's set on all the tables, as I'm using the default for all the tables.
>>> But for that particular table there are 41 SSTables between 60MB and 85MB,
>>> it should only take 4 for the compaction to kick in.
>>>
>>
>> What version of Cassandra are you running?
>>
>> Are they all "live"? Are there pending compactions, or exceptions
>> regarding compactions in your logs?
>>
>>
>>> As this is probably a bug and going back in the mailing list archive, it
>>> seems it's already been reported:
>>>
>>
>> This is a weird statement. Are you saying that you've found it in the
>> mailing list archives? If so, why not paste the threads so those of us who
>> might remember can refer to them?
>>
>>>
>>> - Will it be fixed in 2.1.3?
>>>
>>>
>> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>>
>>
>> =Rob
>>
>>
>
Re: Compaction failing to trigger
Posted by Eric Stevens <mi...@gmail.com>.
@Rob - he's probably referring to the thread titled "Reasons for nodes not
compacting?" where Tyler speculates that the tables are falling below the
cold read threshold for compaction. He speculated it may be a bug. At the
same time in a different thread, Roland had a similar problem, and Tyler's
proposed workaround seemed to work for him.
On Tue, Jan 20, 2015 at 3:35 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Sun, Jan 18, 2015 at 6:06 PM, Flavien Charlon <
> flavien.charlon@gmail.com> wrote:
>
>> It's set on all the tables, as I'm using the default for all the tables.
>> But for that particular table there are 41 SSTables between 60MB and 85MB,
>> it should only take 4 for the compaction to kick in.
>>
>
> What version of Cassandra are you running?
>
> Are they all "live"? Are there pending compactions, or exceptions
> regarding compactions in your logs?
>
>
>> As this is probably a bug and going back in the mailing list archive, it
>> seems it's already been reported:
>>
>
> This is a weird statement. Are you saying that you've found it in the
> mailing list archives? If so, why not paste the threads so those of us who
> might remember can refer to them?
>
>>
>> - Will it be fixed in 2.1.3?
>>
>> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>
>
> =Rob
>
>
Re: Compaction failing to trigger
Posted by Robert Coli <rc...@eventbrite.com>.
On Sun, Jan 18, 2015 at 6:06 PM, Flavien Charlon <fl...@gmail.com>
wrote:
> It's set on all the tables, as I'm using the default for all the tables.
> But for that particular table there are 41 SSTables between 60MB and 85MB,
> it should only take 4 for the compaction to kick in.
>
What version of Cassandra are you running?
Are they all "live"? Are there pending compactions, or exceptions regarding
compactions in your logs?
> As this is probably a bug and going back in the mailing list archive, it
> seems it's already been reported:
>
This is a weird statement. Are you saying that you've found it in the
mailing list archives? If so, why not paste the threads so those of us who
might remember can refer to them?
>
> - Will it be fixed in 2.1.3?
>
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
=Rob
Re: Compaction failing to trigger
Posted by Flavien Charlon <fl...@gmail.com>.
Thanks Roland. Good to know, I will try that. Do you know the JIRA ticket
number of that bug?
Thanks,
Flavien
On 19 January 2015 at 06:15, Roland Etzenhammer <r....@t-online.de>
wrote:
> Hi Flavien,
>
> I hit some problem with minor compations recently (just some days ago) -
> but with many more tables. In my case compactions got not triggered, you
> can check this with nodetool compactionstats.
>
> Reason for me was that those minor compactions did not get triggered since
> there were almost no reads on that tables. Setting 'cold_reads_to_omit' to
> 0 did the job for me:
>
> ALTER TABLE <tablename> WITH compaction = {'class':
> 'SizeTieredCompactionStrategy', 'min_threshold': '4', 'max_threshold':
> '32', 'cold_reads_to_omit': 0.0};
>
> Credits to Tyler and Eric for the pointers.
>
> Cheers,
> Roland
>
Re: Compaction failing to trigger
Posted by Roland Etzenhammer <r....@t-online.de>.
Hi Flavien,
I hit some problem with minor compations recently (just some days ago) -
but with many more tables. In my case compactions got not triggered, you
can check this with nodetool compactionstats.
Reason for me was that those minor compactions did not get triggered
since there were almost no reads on that tables. Setting
'cold_reads_to_omit' to 0 did the job for me:
ALTER TABLE <tablename> WITH compaction = {'class': 'SizeTieredCompactionStrategy', 'min_threshold': '4', 'max_threshold': '32', 'cold_reads_to_omit': 0.0};
Credits to Tyler and Eric for the pointers.
Cheers,
Roland
Re: Compaction failing to trigger
Posted by Flavien Charlon <fl...@gmail.com>.
It's set on all the tables, as I'm using the default for all the tables.
But for that particular table there are 41 SSTables between 60MB and 85MB,
it should only take 4 for the compaction to kick in.
As this is probably a bug and going back in the mailing list archive, it
seems it's already been reported:
- Is there a workaround?
- What is the JIRA ticket number?
- Will it be fixed in 2.1.3?
Thanks
Flavien
On 19 January 2015 at 01:23, 严超 <ya...@gmail.com> wrote:
> Seems like Size Tier Compaction is based on table, which table did you
> set the compaction strategy?
> A minor compaction does not involve all the tables in a keyspace.
> Ref:
>
> http://datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_configure_compaction_t.html
>
> http://datastax.com/documentation/cql/3.1/cql/cql_reference/tabProp.html?scroll=tabProp__moreCompaction
>
> *Best Regards！*
>
>
> *Chao Yan--------------**My twitter：Andy Yan @yanchao727
> <https://twitter.com/yanchao727>*
>
>
> *My Weibo：http://weibo.com/herewearenow
> <http://weibo.com/herewearenow>--------------*
>
> 2015-01-19 3:51 GMT+08:00 Flavien Charlon <fl...@gmail.com>:
>
>> Hi,
>>
>> I am using Size Tier Compaction (Cassandra 2.1.2). Minor compaction is
>> not triggering even though it should. See the SSTables on disk:
>> http://pastebin.com/PSwZ5mrT
>>
>> You can see that we have 41 SSTable between 60MB and 85MB, which should
>> trigger compaction unless I am missing something.
>>
>> Is that a bug?
>>
>> Thanks,
>> Flavien
>>
>
>
Re: Compaction failing to trigger
Posted by 严超 <ya...@gmail.com>.
Seems like Size Tier Compaction is based on table, which table did you set
the compaction strategy?
A minor compaction does not involve all the tables in a keyspace.
Ref:
http://datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_configure_compaction_t.html
http://datastax.com/documentation/cql/3.1/cql/cql_reference/tabProp.html?scroll=tabProp__moreCompaction
*Best Regards！*
*Chao Yan--------------**My twitter：Andy Yan @yanchao727
<https://twitter.com/yanchao727>*
*My Weibo：http://weibo.com/herewearenow
<http://weibo.com/herewearenow>--------------*
2015-01-19 3:51 GMT+08:00 Flavien Charlon <fl...@gmail.com>:
> Hi,
>
> I am using Size Tier Compaction (Cassandra 2.1.2). Minor compaction is not
> triggering even though it should. See the SSTables on disk:
> http://pastebin.com/PSwZ5mrT
>
> You can see that we have 41 SSTable between 60MB and 85MB, which should
> trigger compaction unless I am missing something.
>
> Is that a bug?
>
> Thanks,
> Flavien
>