You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Yang <te...@gmail.com> on 2011/10/19 01:37:28 UTC

commitlog replay extremely slow?

I updated to 71ba6998b504966690e099c03e04f9876dc1060e  on github 1.0.0
branch HEAD,

now the new code seems to be very slow in commitlog replay, it takes
20minutes to recover about 500MB of commit logs, while
previously this takes roughly 1--2 minutes.


is the official 1.0.0 release built on
4f39d3e52f82d060bf96c2be0df6ff6782bc48e5 ?
those changes after this do not sound immediately related to the
possible issue I'm seeing


Thanks
Yang

Re: commitlog replay extremely slow?

Posted by Yang <te...@gmail.com>.

jstack shows that all the mutation stages are blocked on a synchronization:


"MutationStage:1" prio=10 tid=0x00002aaabc01b000 nid=0x27b8 waiting
for monitor entry [0x0000000042048000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.cassandra.db.Table.apply(Table.java:439)
        - waiting to lock <0x000000073caa1630> (a java.lang.Object)
        at org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:337)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)




Table.java: 439

                synchronized (indexLockFor(mutation.key()))
                {
                    ColumnFamily oldIndexedColumns = null;
                    if (mutatedIndexedColumns != null)



it's a bit weird, since I checked that indexLockFor() does return
distinct objects from the array of 4096 objects. but every time I do a
jstack, it shows them in this state, what are they blocking on ??



On Tue, Oct 18, 2011 at 4:37 PM, Yang <te...@gmail.com> wrote:
> I updated to 71ba6998b504966690e099c03e04f9876dc1060e  on github 1.0.0
> branch HEAD,
>
> now the new code seems to be very slow in commitlog replay, it takes
> 20minutes to recover about 500MB of commit logs, while
> previously this takes roughly 1--2 minutes.
>
>
> is the official 1.0.0 release built on
> 4f39d3e52f82d060bf96c2be0df6ff6782bc48e5 ?
> those changes after this do not sound immediately related to the
> possible issue I'm seeing
>
>
> Thanks
> Yang
>