You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Samarth Gahire (Updated) (JIRA)" <ji...@apache.org> on 2012/03/06 15:28:57 UTC
[jira] [Updated] (CASSANDRA-4005) After running scrub , while major
compaction getting exception FileNotFound with (too many open files)
[ https://issues.apache.org/jira/browse/CASSANDRA-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Samarth Gahire updated CASSANDRA-4005:
--------------------------------------
Description:
I was unable to trigger compaction on one of the column families so I ran scrub on that CF.
After scrub when I tried to run compaction I got following error.
{code}
Error occured during compaction
java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.db.compaction.CompactionManager.performMajor(CompactionManager.java:277)
at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1762)
at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1358)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOError: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:660)
at org.apache.cassandra.db.compaction.CompactionIterator.getCollatingIterator(CompactionIterator.java:92)
at org.apache.cassandra.db.compaction.CompactionIterator.<init>(CompactionIterator.java:68)
at org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:552)
at org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:506)
at org.apache.cassandra.db.compaction.CompactionManager$4.call(CompactionManager.java:319)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:57)
... 11 more
{code}
We are using cassndra-0.8.2 and I have set the ulimit to 1048576 on Ubuntu Lucid system.
Also getting the same error while scrub.
was:
I was unable to trigger compaction on one of the column families so I ran scrub on that CF.
After scrub when I tried to run compaction I got following error.
{code}
Error occured during compaction
java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.db.compaction.CompactionManager.performMajor(CompactionManager.java:277)
at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1762)
at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1358)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOError: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:660)
at org.apache.cassandra.db.compaction.CompactionIterator.getCollatingIterator(CompactionIterator.java:92)
at org.apache.cassandra.db.compaction.CompactionIterator.<init>(CompactionIterator.java:68)
at org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:552)
at org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:506)
at org.apache.cassandra.db.compaction.CompactionManager$4.call(CompactionManager.java:319)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:57)
... 11 more
{code}
We are using cassndra-0.8.2 and I have set the ulimit to 1048576 on Ubuntu Lucid system.
> After running scrub , while major compaction getting exception FileNotFound with (too many open files)
> ------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4005
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.2
> Reporter: Samarth Gahire
> Priority: Minor
> Labels: compaction, exception, scrub
> Fix For: 0.8.11
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was unable to trigger compaction on one of the column families so I ran scrub on that CF.
> After scrub when I tried to run compaction I got following error.
> {code}
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.io.IOError: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.cassandra.db.compaction.CompactionManager.performMajor(CompactionManager.java:277)
> at org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1762)
> at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1358)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOError: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:61)
> at org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:660)
> at org.apache.cassandra.db.compaction.CompactionIterator.getCollatingIterator(CompactionIterator.java:92)
> at org.apache.cassandra.db.compaction.CompactionIterator.<init>(CompactionIterator.java:68)
> at org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:552)
> at org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:506)
> at org.apache.cassandra.db.compaction.CompactionManager$4.call(CompactionManager.java:319)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /mnt2/var/lib/cassandra/data/AudienceNetwork/Audience-g-9898-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:57)
> ... 11 more
> {code}
> We are using cassndra-0.8.2 and I have set the ulimit to 1048576 on Ubuntu Lucid system.
> Also getting the same error while scrub.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira