Commitlog still replaying after drain && shutdown

[Dan Kinder <dk...@turnitin.com>]

Hi all,

To quote Sebastian Estevez in one recent thread: "You said you ran a
nodetool drain before the restart, but your logs show commitlogs replayed.
That does not add up..." The docs seem to generally agree with this: if you
did `nodetool drain` before restarting your node there shouldn't be any
commitlogs.

But my experience has been that if I do `nodetool drain`, I need to wait at
least 30-60 seconds after it has finished if I really want no commitlog
replay on restart. If I restart immediately (or even 10-20s later) then it
replays plenty. (This was true on 2.X and is still true on 2.1.7 for me.)

Is this unusual or the same thing others see? Is `nodetool drain` really
supposed to wait until all memtables are flushed and commitlogs are deleted
before it returns?

Thanks,
-dan

Re: Commitlog still replaying after drain && shutdown

[Robert Coli <rc...@eventbrite.com>]

On Tue, Jun 30, 2015 at 11:59 AM, Dan Kinder <dk...@turnitin.com> wrote:

> Is this unusual or the same thing others see? Is `nodetool drain` really
> supposed to wait until all memtables are flushed and commitlogs are deleted
> before it returns?
>

"nodetool drain" *should* work the way you suggest - if one runs it before
shutdown and waits for the "DRAINED" message in the log, one should get no
commitlog replay on start.

In practice, this has historically not-worked at various times.

https://issues.apache.org/jira/browse/CASSANDRA-4446
https://issues.apache.org/jira/browse/CASSANDRA-5911 /
https://issues.apache.org/jira/browse/CASSANDRA-3578 (marked fixed in 2.1
beta)

It's pretty harmless to over-replay in most cases, one is most-hosed if one
is using counters. But if you can repro in 2.1.7 by restarting at any time
after the DRAINED message, you should file an issue on issues.apache.org
and reply here to let the list know the URL for it.

=Rob