You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by yang yang <ya...@yahoo.com.cn> on 2011/09/07 20:01:17 UTC
weird freeze
my cassandra server (from github source head, plus I added some code that just
calls thrift.CassandraServer.batch_mutate() in the same JVM ) runs fine under
heavy load for about 5 hours, then it froze,
when I checked the jstack, all the mutation stages are blocked on
Table.switchlock.ReadLock().lock(),
and the corresponding writeLock is being held by Table.maybeSwitchMemlock(),
which is blocked on CommitLog.instance.getContext(), which is blocked on
get()'ting from a future.
"COMMIT-LOG-WRITER" prio=10 tid=0x00002aaab4a6c000 nid=0x66df waiting on
condition [0x00000000428d6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.$$YJP$$park(Native Method)
- parking to wait for <0x00000007f72bb5f0> (a
java.util.concurrent.FutureTask$Sync)
at sun.misc.Unsafe.park(Unsafe.java)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.db.commitlog.CommitLog.getContext(CommitLog.java:388)
at
org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:661)
at
org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:733)
at
org.apache.cassandra.db.commitlog.CommitLog.createNewSegment(CommitLog.java:575)
at
org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:82)
at
org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:598)
at
org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:662)
I tried to search for the thread that does the getting from future in commitLog,
but it's not present. anyone has an idea why the CommitLog.instance.getContext()
is not returning? the getContext() Callable is not yet scheduled by the
CommitLog executor?
Thanks a lot
Yang
Re: weird freeze
Posted by Yang <te...@gmail.com>.
ok, found this is likely due to GC... I'm seeing full GC that runs
20seconds without actually removing anything:
35240.526: [Full GC [PSYoungGen: 1760704K->1668729K(1848128K)]
[PSOldGen: 4095999K->4095999K(4096000K)] 5856703K->5764729K(5944128K)
[PSPermGen: 24885K->24871K(25152K)], 20.6780790 secs] [Times:
user=20.60 sys=0.00, real=20.68 secs]
I'm trying to do a MAT anaysis now
On Wed, Sep 7, 2011 at 11:01 AM, yang yang <ya...@yahoo.com.cn> wrote:
> my cassandra server (from github source head, plus I added some code that just
> calls thrift.CassandraServer.batch_mutate() in the same JVM ) runs fine under
> heavy load for about 5 hours, then it froze,
>
> when I checked the jstack, all the mutation stages are blocked on
> Table.switchlock.ReadLock().lock(),
> and the corresponding writeLock is being held by Table.maybeSwitchMemlock(),
> which is blocked on CommitLog.instance.getContext(), which is blocked on
> get()'ting from a future.
>
>
> "COMMIT-LOG-WRITER" prio=10 tid=0x00002aaab4a6c000 nid=0x66df waiting on
> condition [0x00000000428d6000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.$$YJP$$park(Native Method)
> - parking to wait for <0x00000007f72bb5f0> (a
> java.util.concurrent.FutureTask$Sync)
> at sun.misc.Unsafe.park(Unsafe.java)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.db.commitlog.CommitLog.getContext(CommitLog.java:388)
> at
> org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:661)
>
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:733)
> at
> org.apache.cassandra.db.commitlog.CommitLog.createNewSegment(CommitLog.java:575)
> at
> org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:82)
> at
> org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:598)
>
> at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
>
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:662)
>
>
>
> I tried to search for the thread that does the getting from future in commitLog,
> but it's not present. anyone has an idea why the CommitLog.instance.getContext()
> is not returning? the getContext() Callable is not yet scheduled by the
> CommitLog executor?
>
>
> Thanks a lot
> Yang