disk full and COMMIT-LOG-WRITER ?

[Yang <te...@gmail.com>]

I found the reason of my server freeze:

COMMIT-LOG-WRITER thread is gone, dead, so the blocking queue in
PeriodicCommitLogExecutorService is full, then all mutationStage jobs
are stuck on the mutations flushing.

the COMMIT-LOG-WRITER thread died because at one time the disk was full,
I cleaned up the disk space (not deleting cassandra files, but other files),
but then since the thread is gone, system is still stuck.
so I had to restart the server.


is it better to let the WRITER thread handle file system exceptions or
let it die? granted letting disk go full is not
a good practice, but letting the system proceed after disk is freed
seems a more natural expectation.

Thanks
Yang

Re: disk full and COMMIT-LOG-WRITER ?

[Yang <te...@gmail.com>]

ok, found past discussions:

http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/cassandra-server-disk-full-td6560725.html

On Thu, Sep 8, 2011 at 11:00 PM, Yang <te...@gmail.com> wrote:
> I found the reason of my server freeze:
>
> COMMIT-LOG-WRITER thread is gone, dead, so the blocking queue in
> PeriodicCommitLogExecutorService is full, then all mutationStage jobs
> are stuck on the mutations flushing.
>
> the COMMIT-LOG-WRITER thread died because at one time the disk was full,
> I cleaned up the disk space (not deleting cassandra files, but other files),
> but then since the thread is gone, system is still stuck.
> so I had to restart the server.
>
>
> is it better to let the WRITER thread handle file system exceptions or
> let it die? granted letting disk go full is not
> a good practice, but letting the system proceed after disk is freed
> seems a more natural expectation.
>
> Thanks
> Yang
>