Logs of commitlog files

[Donald Smith <Do...@audiencescience.com>]

1. With cassandra 2.0.6, we have 547G of files in /var/lib/commitlog/.  I started a "nodetool flush" 65 minutes ago; it's still running.  The 17536 commitlog files have been created in the last 3 days.  (The node has 2.1T of sstables data in /var/lib/cassandra/data/.  This is in staging, not prod.) Why so many commit logs?  Here are our commitlog-related settings in cassandra.yaml:

commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
# The size of the individual commitlog file segments.  A commitlog
# archiving commitlog segments (see commitlog_archiving.properties),
commitlog_segment_size_in_mb: 32
# Total space to use for commitlogs.  Since commitlog segments are
# segment and remove it.  So a small total commitlog space will tend
# commitlog_total_space_in_mb: 4096

Maybe we should set commitlog_total_space_in_mb to something other than the default. According to OpsCenter, commitlog_total_space_in_mb is "None".    But it seems odd that there'd be so many commit logs.

The node is under heavy write load.   There are about 2900 compactions pending.

We are NOT archiving commitlogs, via commitlog_archiving.properties.

BTW, the documentation for nodetool<http://wiki.apache.org/cassandra/NodeTool> says:
Flush

Flushes memtables (in memory) to SSTables (on disk), which also enables CommitLog<http://wiki.apache.org/cassandra/CommitLog> segments to be deleted.
But even after doing a flush, the /var/lib/commitlog dir still has 1G of files, even after waiting 30  minutes.  Each file is 32M in size, plus or minus a few bytes.  I tried this on other clusters, with much smaller amounts of data.   Even restarting Cassandra doesn't help.

I surmise that the 1GB of commit logs are normal: they probably allocate that space as a workspace.


Thanks,  Don

Donald A. Smith | Senior Software Engineer
P: 425.201.3900 x 3866
C: (206) 819-5965
F: (646) 443-2333
donalds@AudienceScience.com<ma...@AudienceScience.com>

[AudienceScience]