You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Andreas Schnitzerling (JIRA)" <ji...@apache.org> on 2013/12/09 18:20:08 UTC
[jira] [Commented] (CASSANDRA-6462) cleanup ClassCastException
[ https://issues.apache.org/jira/browse/CASSANDRA-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843336#comment-13843336 ]
Andreas Schnitzerling commented on CASSANDRA-6462:
--------------------------------------------------
Cleanup on that node not working anymore. There are already problems after restart of the cassandra service. Any ideas how to fix? If u don't have idea as well I will try to repair, but maybe cannot reproduce anymore.
{panel:title=system.log}
ERROR [CompactionExecutor:25] 2013-12-09 17:50:14,932 CassandraDaemon.java (line 187) Exception in thread Thread[CompactionExecutor:25,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1dbf287 rejected from org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@10f8a54[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 780]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor.submit(Unknown Source)
at org.apache.cassandra.io.sstable.SSTableDeletingTask.schedule(SSTableDeletingTask.java:66)
at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1105)
at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:388)
at org.apache.cassandra.db.DataTracker.postReplace(DataTracker.java:353)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:347)
at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:252)
at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1078)
at org.apache.cassandra.db.compaction.CompactionTask.replaceCompactedSSTables(CompactionTask.java:296)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:242)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
ERROR [ValidationExecutor:1] 2013-12-09 17:53:16,411 Validator.java (line 242) Failed creating a merkle tree for [repair #65efe6e0-60f2-11e3-ac47-eb1c24a59bb8 on nieste/nfiles, (3837863087520363054,3847126934337600104]], /10.9.9.240 (see log for details)
ERROR [ValidationExecutor:1] 2013-12-09 17:53:16,421 CassandraDaemon.java (line 187) Exception in thread Thread[ValidationExecutor:1,1,main]
FSWriteError in D:\Programme\cassandra\data\nieste\nfiles\snapshots\65efe6e0-60f2-11e3-ac47-eb1c24a59bb8\nieste-nfiles-jb-19878-Index.db
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:120)
at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:382)
at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:378)
at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:416)
at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1801)
at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:810)
at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.FileSystemException: D:\Programme\cassandra\data\nieste\nfiles\snapshots\65efe6e0-60f2-11e3-ac47-eb1c24a59bb8\nieste-nfiles-jb-19878-Index.db: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
at java.nio.file.Files.delete(Unknown Source)
at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:116)
... 11 more
ERROR [CompactionExecutor:3] 2013-12-09 17:54:16,810 CassandraDaemon.java (line 187) Exception in thread Thread[CompactionExecutor:3,1,RMI Runtime]
java.lang.ClassCastException: org.apache.cassandra.io.sstable.SSTableReader$EmptyCompactionScanner cannot be cast to org.apache.cassandra.io.sstable.SSTableScanner
at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:563)
at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:62)
at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:274)
at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:222)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
{panel}
> cleanup ClassCastException
> --------------------------
>
> Key: CASSANDRA-6462
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6462
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: Windows 7 / Java 1.7.0.25
> Reporter: Andreas Schnitzerling
> Labels: cast, cleanup, compaction, concurrency, newbie, nodetool, remote
> Fix For: 2.0.4
>
>
> I enlarged the cluseter from 4 to 8 nodes. During cleaning up the "old" nodes with "nodetool cleanup" it breaks up with exception. I started cleanup from a different computer to manage them sequentially.
> {panel:title=cmd.exe}
> Error occurred during cleanup
> java.util.concurrent.ExecutionException: java.lang.ClassCastException: org.apach
> e.cassandra.io.sstable.SSTableReader$EmptyCompactionScanner cannot be cast to or
> g.apache.cassandra.io.sstable.SSTableScanner
> at java.util.concurrent.FutureTask.report(Unknown Source)
> at java.util.concurrent.FutureTask.get(Unknown Source)
> at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTabl
> eOperation(CompactionManager.java:227)
> at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(C
> ompactionManager.java:265)
> at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilySt
> ore.java:1054)
> at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(Stor
> ageService.java:2038)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at sun.reflect.misc.Trampoline.invoke(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at sun.reflect.misc.MethodUtil.invoke(Unknown Source)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown So
> urce)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown So
> urce)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown
> Source)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Sou
> rce)
> at javax.management.remote.rmi.RMIConnectionImpl.access$300(Unknown Sour
> ce)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run
> (Unknown Source)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(U
> nknown Source)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
> at sun.rmi.transport.Transport$1.run(Unknown Source)
> at sun.rmi.transport.Transport$1.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Unknown Source)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Sou
> rce)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Sour
> ce)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.ClassCastException: org.apache.cassandra.io.sstable.SSTable
> Reader$EmptyCompactionScanner cannot be cast to org.apache.cassandra.io.sstable.
> SSTableScanner
> at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompact
> ion(CompactionManager.java:563)
> at org.apache.cassandra.db.compaction.CompactionManager.access$400(Compa
> ctionManager.java:62)
> at org.apache.cassandra.db.compaction.CompactionManager$5.perform(Compac
> tionManager.java:274)
> at org.apache.cassandra.db.compaction.CompactionManager$2.call(Compactio
> nManager.java:222)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> ... 3 more
> {panel}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)