You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2010/01/13 21:37:54 UTC
[jira] Updated: (CASSANDRA-694) Failure to flush commit log
[ https://issues.apache.org/jira/browse/CASSANDRA-694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-694:
-------------------------------------
Attachment: 694-0.5.txt
assumption that all single-threaded executors have an unbounded queue is no longer valid. this patch provides a policy for dealing with single thread executors w/ a full queue.
> Failure to flush commit log
> ---------------------------
>
> Key: CASSANDRA-694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-694
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.5
> Environment: Linux 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 GNU/Linux, ec2 small instance
> Reporter: Ryan Daum
> Attachments: 694-0.5.txt
>
>
> The following exception occurs consistently on at least node (note did not occur on other same-configured nodes) during startup:
> INFO - Replaying /var/lib/cassandra/commitlog/CommitLog-1262855754427.log, /var/lib/cassandra/commitlog/CommitLog-1262832689989.log, /var/lib/cassandra/commitlog/CommitLog-1262885833186.log, /var/lib/cassandra/commitlog/CommitLog-1262900845019.log, /var/lib/cassandra/commitlog/CommitLog-1262913267844.log, /var/lib/cassandra/commitlog/CommitLog-1262927898170.log, /var/lib/cassandra/commitlog/CommitLog-1262961421039.log, /var/lib/cassandra/commitlog/CommitLog-1262977175175.log, /var/lib/cassandra/commitlog/CommitLog-1262989588783.log, /var/lib/cassandra/commitlog/CommitLog-1263000573676.log, /var/lib/cassandra/commitlog/CommitLog-1263013691393.log, /var/lib/cassandra/commitlog/CommitLog-1263044706108.log, /var/lib/cassandra/commitlog/CommitLog-1263060004191.log, /var/lib/cassandra/commitlog/CommitLog-1263071446342.log, /var/lib/cassandra/commitlog/CommitLog-1263082950154.log, /var/lib/cassandra/commitlog/CommitLog-1263095400814.log, /var/lib/cassandra/commitlog/CommitLog-1263118331046.log, /var/lib/cassandra/commitlog/CommitLog-1263143402963.log, /var/lib/cassandra/commitlog/CommitLog-1263155294308.log, /var/lib/cassandra/commitlog/CommitLog-1263166154352.log, /var/lib/cassandra/commitlog/CommitLog-1263178359247.log, /var/lib/cassandra/commitlog/CommitLog-1263202112017.log, /var/lib/cassandra/commitlog/CommitLog-1263230932274.log, /var/lib/cassandra/commitlog/CommitLog-1263250726505.log, /var/lib/cassandra/commitlog/CommitLog-1263264159438.log, /var/lib/cassandra/commitlog/CommitLog-1263289964249.log, /var/lib/cassandra/commitlog/CommitLog-1263317974387.log, /var/lib/cassandra/commitlog/CommitLog-1263331989090.log, /var/lib/cassandra/commitlog/CommitLog-1263344147667.log, /var/lib/cassandra/commitlog/CommitLog-1263359751527.log, /var/lib/cassandra/commitlog/CommitLog-1263395707008.log, /var/lib/cassandra/commitlog/CommitLog-1263397833524.log, /var/lib/cassandra/commitlog/CommitLog-1263398736183.log, /var/lib/cassandra/commitlog/CommitLog-1263399753707.log, /var/lib/cassandra/commitlog/CommitLog-1263401667504.log, /var/lib/cassandra/commitlog/CommitLog-1263404640782.log, /var/lib/cassandra/commitlog/CommitLog-1263405827234.log, /var/lib/cassandra/commitlog/CommitLog-1263406901115.log
> INFO - LocationInfo has reached its threshold; switching in a fresh Memtable
> INFO - Enqueuing flush of Memtable(LocationInfo)@25934689
> INFO - HintsColumnFamily has reached its threshold; switching in a fresh Memtable
> INFO - Enqueuing flush of Memtable(HintsColumnFamily)@4766820
> INFO - AdXRequestStatistics has reached its threshold; switching in a fresh Memtable
> INFO - Enqueuing flush of Memtable(AdXRequestStatistics)@21521158
> INFO - TokenGoogleIDCF has reached its threshold; switching in a fresh Memtable
> INFO - Enqueuing flush of Memtable(TokenGoogleIDCF)@22889075
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)
> Caused by: java.lang.AssertionError: Blocking serialized executor is not yet implemented
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:84)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78)
> at org.apache.cassandra.db.ColumnFamilyStore.submitFlush(ColumnFamilyStore.java:1045)
> at org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:395)
> at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:448)
> at org.apache.cassandra.db.Table.flush(Table.java:464)
> at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:397)
> at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90)
> at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:135)
> ... 5 more
> And the same exception occurs intermittently on other node (running) nodes during 'nodeprobe flush':
> root@domU-12-31-38-00-26-31:~# nodeprobe -host localhost -port 8080 flush Logger
> Exception in thread "main" java.lang.AssertionError: Blocking serialized executor is not yet implemented
> at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:84)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78)
> at org.apache.cassandra.db.ColumnFamilyStore.submitFlush(ColumnFamilyStore.java:1045)
> at org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:395)
> at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:448)
> at org.apache.cassandra.service.StorageService.forceTableFlush(StorageService.java:984)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1359)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.