Some commit logs not deleting

[Chad Johnson <ch...@gmail.com>]

Hello,

We are running Cassandra 0.7.5 on a 15 node cluster, RF=3. We are having a
problem where some commit logs do not get deleted. Our write load generates
a new commit log about every two to three minutes. On average, one commit
log an hour is not deleted. Without draining, deleting the remaining commit
log files and restarting each node in the cluster, the commit log partition
will fill up. We do one thing with our cluster that is probably not very
common. We make schema changes four times per day. We cycle column families
by dropping old column families for old data we don't care about any longer
and creating new ones for new data.

Is anybody else see this problem? I can assume that the dirty bit for those
commit logs are still set, but why? How can I determine what CF memtable is
still dirty?

Please let me know if there is additional information I can provide and
thanks for your help.

Chad

Re: Some commit logs not deleting

[Sylvain Lebresne <sy...@datastax.com>]

This is likely due to
https://issues.apache.org/jira/browse/CASSANDRA-2829. Basically, if
you have a column family on which you stop to write suddenly forever
(which will be the case if you drop a cf), one commit log could get
retained forever (forever meaning "until next restart of the node" in
that context). The patch is targeted for 0.8.3 and I see no good
reason why it wouldn't make it in, but I don't think we'll push it in
the 0.7 series.

--
Sylvain

On Mon, Jul 25, 2011 at 4:48 AM, Chad Johnson <ch...@gmail.com> wrote:
> Hello,
>
> We are running Cassandra 0.7.5 on a 15 node cluster, RF=3. We are having a
> problem where some commit logs do not get deleted. Our write load generates
> a new commit log about every two to three minutes. On average, one commit
> log an hour is not deleted. Without draining, deleting the remaining commit
> log files and restarting each node in the cluster, the commit log partition
> will fill up. We do one thing with our cluster that is probably not very
> common. We make schema changes four times per day. We cycle column families
> by dropping old column families for old data we don't care about any longer
> and creating new ones for new data.
>
> Is anybody else see this problem? I can assume that the dirty bit for those
> commit logs are still set, but why? How can I determine what CF memtable is
> still dirty?
>
> Please let me know if there is additional information I can provide and
> thanks for your help.
>
> Chad
>