You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Mokkapati, Bhargav (Nokia - IN/Chennai)" <bh...@nokia.com> on 2018/01/30 10:30:13 UTC
Commitlogs are filling the Full Disk space and nodes are down
Hi Team,
My Cassandra version : Apache Cassandra 3.0.13
Cassandra nodes are down due to Commitlogs are getting filled up until full disk size.
[cid:image001.jpg@01D399E3.666CF940]
With "Nodetool flush" I didn't see any commitlogs deleted.
Can anyone tell me how to flush the commitlogs without losing data.
Thanks,
Bhargav M
Re: Commitlogs are filling the Full Disk space and nodes are down
Posted by Jeff Jirsa <jj...@gmail.com>.
There's an open bug for users that have offheap memtables and secondary
index - there's at least a few people reporting an error flushing that
blocks future flushes.
If you're seeing that, and use that combo, you may want to switch to
on-heap memtables (or contribute a patch to fix the offheap+2i interaction)
On Tue, Jan 30, 2018 at 9:06 AM, Chris Lohfink <cl...@apple.com> wrote:
> The commitlog growing is often a symptom of a problem. If the memtable
> flush or post flush fails in anyway, the commitlogs will not be
> recycled/deleted and will continue to pool up.
>
> Might want to go back in logs earlier to make sure theres nothing like the
> postmemtable flusher getting a permission error (some tooling creates
> commitlogs so if run by wrong user can create this prooblem), or a memtable
> flush error. You can also check tpstats to see if tasks are queued up in
> postmemtable flusher and jstack to see where the active ones are stuck if
> they are.
>
> Chris
>
> On Jan 30, 2018, at 4:20 AM, Amit Singh <de...@nic.in> wrote:
>
> Hi,
>
> When you actually say nodetool flush, data from memTable goes to disk
> based structure as SStables and side by side , commit logs segments for
> that particular data get written off and its continuous process . May be in
> your case , you can decrease the value of below uncommented property in
> Cassandra.yaml
>
> commitlog_total_space_in_mb
>
> Also this is what is it used for
>
> # Total space to use for commit logs on disk.
> #
> # If space gets above this value, Cassandra will flush every dirty CF
> # in the oldest segment and remove it. So a small total commitlog space
> # will tend to cause more flush activity on less-active columnfamilies.
> #
> # The default value is the smaller of 8192, and 1/4 of the total space
> # of the commitlog volume.
>
>
> *From:* Mokkapati, Bhargav (Nokia - IN/Chennai) [mailto:bhargav.mokkapati@
> nokia.com <bh...@nokia.com>]
> *Sent:* Tuesday, January 30, 2018 4:00 PM
> *To:* user@cassandra.apache.org
> *Subject:* Commitlogs are filling the Full Disk space and nodes are down
>
> Hi Team,
>
> My Cassandra version : Apache Cassandra 3.0.13
>
> Cassandra nodes are down due to Commitlogs are getting filled up until
> full disk size.
>
> <image001.jpg>
>
> With “Nodetool flush” I didn’t see any commitlogs deleted.
>
> Can anyone tell me how to flush the commitlogs without losing data.
>
> Thanks,
> Bhargav M
>
>
>
Re: Commitlogs are filling the Full Disk space and nodes are down
Posted by Chris Lohfink <cl...@apple.com>.
The commitlog growing is often a symptom of a problem. If the memtable flush or post flush fails in anyway, the commitlogs will not be recycled/deleted and will continue to pool up.
Might want to go back in logs earlier to make sure theres nothing like the postmemtable flusher getting a permission error (some tooling creates commitlogs so if run by wrong user can create this prooblem), or a memtable flush error. You can also check tpstats to see if tasks are queued up in postmemtable flusher and jstack to see where the active ones are stuck if they are.
Chris
> On Jan 30, 2018, at 4:20 AM, Amit Singh <de...@nic.in> wrote:
>
> Hi,
>
> When you actually say nodetool flush, data from memTable goes to disk based structure as SStables and side by side , commit logs segments for that particular data get written off and its continuous process . May be in your case , you can decrease the value of below uncommented property in Cassandra.yaml
>
> commitlog_total_space_in_mb
>
> Also this is what is it used for
>
> # Total space to use for commit logs on disk.
> #
> # If space gets above this value, Cassandra will flush every dirty CF
> # in the oldest segment and remove it. So a small total commitlog space
> # will tend to cause more flush activity on less-active columnfamilies.
> #
> # The default value is the smaller of 8192, and 1/4 of the total space
> # of the commitlog volume.
>
>
> From: Mokkapati, Bhargav (Nokia - IN/Chennai) [mailto:bhargav.mokkapati@nokia.com]
> Sent: Tuesday, January 30, 2018 4:00 PM
> To: user@cassandra.apache.org
> Subject: Commitlogs are filling the Full Disk space and nodes are down
>
> Hi Team,
>
> My Cassandra version : Apache Cassandra 3.0.13
>
> Cassandra nodes are down due to Commitlogs are getting filled up until full disk size.
>
> <image001.jpg>
>
> With “Nodetool flush” I didn’t see any commitlogs deleted.
>
> Can anyone tell me how to flush the commitlogs without losing data.
>
> Thanks,
> Bhargav M
RE: Commitlogs are filling the Full Disk space and nodes are down
Posted by Amit Singh <de...@nic.in>.
Hi,
When you actually say nodetool flush, data from memTable goes to disk based
structure as SStables and side by side , commit logs segments for that
particular data get written off and its continuous process . May be in your
case , you can decrease the value of below uncommented property in
Cassandra.yaml
commitlog_total_space_in_mb
Also this is what is it used for
# Total space to use for commit logs on disk.
#
# If space gets above this value, Cassandra will flush every dirty CF
# in the oldest segment and remove it. So a small total commitlog space
# will tend to cause more flush activity on less-active columnfamilies.
#
# The default value is the smaller of 8192, and 1/4 of the total space
# of the commitlog volume.
From: Mokkapati, Bhargav (Nokia - IN/Chennai)
[mailto:bhargav.mokkapati@nokia.com]
Sent: Tuesday, January 30, 2018 4:00 PM
To: user@cassandra.apache.org
Subject: Commitlogs are filling the Full Disk space and nodes are down
Hi Team,
My Cassandra version : Apache Cassandra 3.0.13
Cassandra nodes are down due to Commitlogs are getting filled up until full
disk size.
With "Nodetool flush" I didn't see any commitlogs deleted.
Can anyone tell me how to flush the commitlogs without losing data.
Thanks,
Bhargav M