You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Sean Bridges <se...@gmail.com> on 2010/06/23 23:26:42 UTC
10 minute cassandra pause
We were running a load test against a single 0.6.2 cassandra node. 24
hours into the test, Cassandra appeared to be nearly frozen for 10
minutes. Our write rate went to almost 0, and we had a large number
of write timeouts. We weren't swapping or gc'ing at the time.
It looks like the problems were caused by our memtables flushing after
24 hours (we have MemtableFlushAfterMinutes=1440). Some of our column
families are written to infrequently so that they don't hit the flush
thresholds in MemtableOperationsInMillions and MemtableThroughputInMB.
After 24 hours we had ~3000 commit log files.
Is this flushing causing Cassandra to become unresponsive? I would
have thought Cassandra could flush in the background without blocking
new writes.
Thanks,
Sean
Re: 10 minute cassandra pause
Posted by Jonathan Ellis <jb...@gmail.com>.
Right, I think there must be some other factor here because neither
the flush nor discard of obsolete commitlog segments block writes.
On Wed, Jun 23, 2010 at 7:05 PM, Sean Bridges <se...@gmail.com> wrote:
> I see about 3000 lines of,
>
> INFO [COMMIT-LOG-WRITER] 2010-06-23 16:40:29,107 CommitLog.java (line
> 412) Discarding obsolete commit
> log:CommitLogSegment(/data1/cass/commitlog/CommitLog-1277302220723.log)
>
> Then, http://pastebin.com/YQA0mpRG
>
> It's around 16:50 that cassandra writes stop timing out. Some writes
> are getting through during this 10 minutes, but they shouldn't be
> enough to cause the index memtables to flush.
>
> Thanks,
>
> Sean
>
>
>
>
>
>
> On Wed, Jun 23, 2010 at 3:30 PM, Benjamin Black <b...@b3k.us> wrote:
>> Are you seeing any sort of log messages from Cassandra at all?
>>
>> On Wed, Jun 23, 2010 at 2:26 PM, Sean Bridges <se...@gmail.com> wrote:
>>> We were running a load test against a single 0.6.2 cassandra node. 24
>>> hours into the test, Cassandra appeared to be nearly frozen for 10
>>> minutes. Our write rate went to almost 0, and we had a large number
>>> of write timeouts. We weren't swapping or gc'ing at the time.
>>>
>>> It looks like the problems were caused by our memtables flushing after
>>> 24 hours (we have MemtableFlushAfterMinutes=1440). Some of our column
>>> families are written to infrequently so that they don't hit the flush
>>> thresholds in MemtableOperationsInMillions and MemtableThroughputInMB.
>>> After 24 hours we had ~3000 commit log files.
>>>
>>> Is this flushing causing Cassandra to become unresponsive? I would
>>> have thought Cassandra could flush in the background without blocking
>>> new writes.
>>>
>>> Thanks,
>>>
>>> Sean
>>>
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: 10 minute cassandra pause
Posted by Sean Bridges <se...@gmail.com>.
I see about 3000 lines of,
INFO [COMMIT-LOG-WRITER] 2010-06-23 16:40:29,107 CommitLog.java (line
412) Discarding obsolete commit
log:CommitLogSegment(/data1/cass/commitlog/CommitLog-1277302220723.log)
Then, http://pastebin.com/YQA0mpRG
It's around 16:50 that cassandra writes stop timing out. Some writes
are getting through during this 10 minutes, but they shouldn't be
enough to cause the index memtables to flush.
Thanks,
Sean
On Wed, Jun 23, 2010 at 3:30 PM, Benjamin Black <b...@b3k.us> wrote:
> Are you seeing any sort of log messages from Cassandra at all?
>
> On Wed, Jun 23, 2010 at 2:26 PM, Sean Bridges <se...@gmail.com> wrote:
>> We were running a load test against a single 0.6.2 cassandra node. 24
>> hours into the test, Cassandra appeared to be nearly frozen for 10
>> minutes. Our write rate went to almost 0, and we had a large number
>> of write timeouts. We weren't swapping or gc'ing at the time.
>>
>> It looks like the problems were caused by our memtables flushing after
>> 24 hours (we have MemtableFlushAfterMinutes=1440). Some of our column
>> families are written to infrequently so that they don't hit the flush
>> thresholds in MemtableOperationsInMillions and MemtableThroughputInMB.
>> After 24 hours we had ~3000 commit log files.
>>
>> Is this flushing causing Cassandra to become unresponsive? I would
>> have thought Cassandra could flush in the background without blocking
>> new writes.
>>
>> Thanks,
>>
>> Sean
>>
>
Re: 10 minute cassandra pause
Posted by Benjamin Black <b...@b3k.us>.
Are you seeing any sort of log messages from Cassandra at all?
On Wed, Jun 23, 2010 at 2:26 PM, Sean Bridges <se...@gmail.com> wrote:
> We were running a load test against a single 0.6.2 cassandra node. 24
> hours into the test, Cassandra appeared to be nearly frozen for 10
> minutes. Our write rate went to almost 0, and we had a large number
> of write timeouts. We weren't swapping or gc'ing at the time.
>
> It looks like the problems were caused by our memtables flushing after
> 24 hours (we have MemtableFlushAfterMinutes=1440). Some of our column
> families are written to infrequently so that they don't hit the flush
> thresholds in MemtableOperationsInMillions and MemtableThroughputInMB.
> After 24 hours we had ~3000 commit log files.
>
> Is this flushing causing Cassandra to become unresponsive? I would
> have thought Cassandra could flush in the background without blocking
> new writes.
>
> Thanks,
>
> Sean
>