You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rana Aich <ai...@gmail.com> on 2010/10/06 20:51:01 UTC
Strange Behavior : Commitlog data is not flushed
Hello Experts,
I see a queer behavior from on of the Cassandra nodes in my cluster where
the data is not flushed off Commitlogs and the Commitlog file grows in
number. I was inserting the data into the cluster and since yesterday this
node had more than 900 commitlog files.
-rw-r--r-- 1 dev dev 134218072 2010-10-05 15:11 CommitLog-1286218976629.log
-rw-r--r-- 1 dev dev 134217909 2010-10-05 15:11 CommitLog-1286316625590.log
-rw-r--r-- 1 dev dev 134217872 2010-10-05 15:15 CommitLog-1286316673050.log
-rw-r--r-- 1 dev dev 134217957 2010-10-05 15:19 CommitLog-1286316917618.log
-rw-r--r-- 1 dev dev 134217764 2010-10-05 15:21 CommitLog-1286317178922.log
-rw-r--r-- 1 dev dev 134217871 2010-10-05 15:24 CommitLog-1286317311372.log
-rw-r--r-- 1 dev dev 134218029 2010-10-05 15:25 CommitLog-1286317446787.log
-rw-r--r-- 1 dev dev 134218065 2010-10-05 15:26 CommitLog-1286317506892.log
-rw-r--r-- 1 dev dev 134218100 2010-10-05 15:27 CommitLog-1286317567927.log
-rw-r--r-- 1 dev dev 134218020 2010-10-05 15:29 CommitLog-1286317634668.log
-rw-r--r-- 1 dev dev 134217922 2010-10-05 15:31 CommitLog-1286317741576.log
-rw-r--r-- 1 dev dev 134217944 2010-10-05 15:32 CommitLog-1286317887655.log
-rw-r--r-- 1 dev dev 134217961 2010-10-05 15:33 CommitLog-1286317968395.log
-rw-r--r-- 1 dev dev 134217926 2010-10-05 15:34 CommitLog-1286318033241.log
-rw-r--r-- 1 dev dev 134218045 2010-10-05 15:36 CommitLog-1286318097727.log
-rw-r--r-- 1 dev dev 134218027 2010-10-05 15:37 CommitLog-1286318190654.log
-rw-r--r-- 1 dev dev 134218060 2010-10-05 15:38 CommitLog-1286318277987.log
-rw-r--r-- 1 dev dev 134218055 2010-10-05 15:41 CommitLog-1286318338561.log
-rw-r--r-- 1 dev dev 134218069 2010-10-05 15:41 CommitLog-1286318461180.log
-rw-r--r-- 1 dev dev 134217791 2010-10-05 15:43 CommitLog-1286318519193.log
-rw-r--r-- 1 dev dev 134218066 2010-10-05 15:44 CommitLog-1286318609960.log
-rw-r--r-- 1 dev dev 134218093 2010-10-05 15:45 CommitLog-1286318692656.log
It seems that commitlog files spawned every minute and when it grew over 900
files the Cassandra node went down.
Now I'm trying to start my node but it has started to read all of those
Commitlog files and takes huge amount of time to restart.
Can anyone throw some light on this unusual behavior. Till yesterday before
starting the data insertion the node had almost 550GB of data. Incidentally
I did stop another node prior to inserting the data because it was 100%
full. I'm using RandomPartitioner with Initial Tokens.
Thanking all in advance.
Rana
Re: Strange Behavior : Commitlog data is not flushed
Posted by Jonathan Ellis <jb...@gmail.com>.
Commitlog segments remain until all the data in them has been flushed.
Reduce MemtableFlushAfterMinutes.
If I had to guess without your error log why the node went down, I
would guess you exceeded the open file handle allowance. You can
increase that with the standard ulimit or /etc/security/limits.conf
methods.
On Wed, Oct 6, 2010 at 1:51 PM, Rana Aich <ai...@gmail.com> wrote:
> Hello Experts,
> I see a queer behavior from on of the Cassandra nodes in my cluster where
> the data is not flushed off Commitlogs and the Commitlog file grows in
> number. I was inserting the data into the cluster and since yesterday this
> node had more than 900 commitlog files.
> -rw-r--r-- 1 dev dev 134218072 2010-10-05 15:11 CommitLog-1286218976629.log
> -rw-r--r-- 1 dev dev 134217909 2010-10-05 15:11 CommitLog-1286316625590.log
> -rw-r--r-- 1 dev dev 134217872 2010-10-05 15:15 CommitLog-1286316673050.log
> -rw-r--r-- 1 dev dev 134217957 2010-10-05 15:19 CommitLog-1286316917618.log
> -rw-r--r-- 1 dev dev 134217764 2010-10-05 15:21 CommitLog-1286317178922.log
> -rw-r--r-- 1 dev dev 134217871 2010-10-05 15:24 CommitLog-1286317311372.log
> -rw-r--r-- 1 dev dev 134218029 2010-10-05 15:25 CommitLog-1286317446787.log
> -rw-r--r-- 1 dev dev 134218065 2010-10-05 15:26 CommitLog-1286317506892.log
> -rw-r--r-- 1 dev dev 134218100 2010-10-05 15:27 CommitLog-1286317567927.log
> -rw-r--r-- 1 dev dev 134218020 2010-10-05 15:29 CommitLog-1286317634668.log
> -rw-r--r-- 1 dev dev 134217922 2010-10-05 15:31 CommitLog-1286317741576.log
> -rw-r--r-- 1 dev dev 134217944 2010-10-05 15:32 CommitLog-1286317887655.log
> -rw-r--r-- 1 dev dev 134217961 2010-10-05 15:33 CommitLog-1286317968395.log
> -rw-r--r-- 1 dev dev 134217926 2010-10-05 15:34 CommitLog-1286318033241.log
> -rw-r--r-- 1 dev dev 134218045 2010-10-05 15:36 CommitLog-1286318097727.log
> -rw-r--r-- 1 dev dev 134218027 2010-10-05 15:37 CommitLog-1286318190654.log
> -rw-r--r-- 1 dev dev 134218060 2010-10-05 15:38 CommitLog-1286318277987.log
> -rw-r--r-- 1 dev dev 134218055 2010-10-05 15:41 CommitLog-1286318338561.log
> -rw-r--r-- 1 dev dev 134218069 2010-10-05 15:41 CommitLog-1286318461180.log
> -rw-r--r-- 1 dev dev 134217791 2010-10-05 15:43 CommitLog-1286318519193.log
> -rw-r--r-- 1 dev dev 134218066 2010-10-05 15:44 CommitLog-1286318609960.log
> -rw-r--r-- 1 dev dev 134218093 2010-10-05 15:45 CommitLog-1286318692656.log
> It seems that commitlog files spawned every minute and when it grew over 900
> files the Cassandra node went down.
> Now I'm trying to start my node but it has started to read all of those
> Commitlog files and takes huge amount of time to restart.
> Can anyone throw some light on this unusual behavior. Till yesterday before
> starting the data insertion the node had almost 550GB of data. Incidentally
> I did stop another node prior to inserting the data because it was 100%
> full. I'm using RandomPartitioner with Initial Tokens.
> Thanking all in advance.
> Rana
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com