Commit log grows and grows

[Fredrik Stigbäck <fr...@sitevision.se>]

I have a problem or at least something that I have missed to
understand regarding commit log removal.
I'm running cassandra 1.0.1 but have seen the same "thing" in 1.0.
I thought that when a node is flushed or drained, it's memtable is
flushed to disk and the commit log is removed and new commit logs are
created as write requests comes in.
I've setup a small test, adding column family data through
cassandra-cli then running both flush and drain but the commit log
isn't removed.
There is only one commit log in the commit log directory.
I've turned on debug and the log always reports "Not safe to delete...":

DEBUG [COMMIT-LOG-WRITER] 2011-11-01 20:51:19,195 CommitLog.java (line
458) discard completed log segments for
ReplayPosition(segmentId=1320176934665, position=340), column family
7.
DEBUG [COMMIT-LOG-WRITER] 2011-11-01 20:51:19,196 CommitLog.java (line
497) Not safe to delete commit log
CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1320176934665.log);
dirty is ; hasNext: false

Could someone please clarify the relationship between commit log
creation, removal/rotation and flush, drain.
Regards
/Fredrik

Re: Commit log grows and grows

[Jonathan Ellis <jb...@gmail.com>]

"hasnext false" means "this is the last commitlog segment, so we'll
keep it around to add new entries too until it fills up at 128MB.

On Tue, Nov 1, 2011 at 3:06 PM, Fredrik Stigbäck
<fr...@sitevision.se> wrote:
> I have a problem or at least something that I have missed to
> understand regarding commit log removal.
> I'm running cassandra 1.0.1 but have seen the same "thing" in 1.0.
> I thought that when a node is flushed or drained, it's memtable is
> flushed to disk and the commit log is removed and new commit logs are
> created as write requests comes in.
> I've setup a small test, adding column family data through
> cassandra-cli then running both flush and drain but the commit log
> isn't removed.
> There is only one commit log in the commit log directory.
> I've turned on debug and the log always reports "Not safe to delete...":
>
> DEBUG [COMMIT-LOG-WRITER] 2011-11-01 20:51:19,195 CommitLog.java (line
> 458) discard completed log segments for
> ReplayPosition(segmentId=1320176934665, position=340), column family
> 7.
> DEBUG [COMMIT-LOG-WRITER] 2011-11-01 20:51:19,196 CommitLog.java (line
> 497) Not safe to delete commit log
> CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1320176934665.log);
> dirty is ; hasNext: false
>
> Could someone please clarify the relationship between commit log
> creation, removal/rotation and flush, drain.
> Regards
> /Fredrik
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

How many indexes to keep? Guidelines

[Maxim Potekhin <po...@bnl.gov>]

As a matter of practice, how many secondary indexes on a CF do you 
usually keep?
What are rules of thumb? Is 10 too many? 100? 1000?

Thanks

Maxim