You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Alex Whittaker (JIRA)" <ji...@apache.org> on 2012/05/03 23:14:48 UTC
[jira] [Created] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Alex Whittaker created CASSANDRA-4218:
-----------------------------------------
Summary: NPE thrown after switching to LeveledCompactionStrategy
Key: CASSANDRA-4218
URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.9, 1.0.5
Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
Reporter: Alex Whittaker
After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
java.lang.NullPointerException
at java.io.File.<init>(File.java:222)
at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267851#comment-13267851 ]
Jonathan Ellis commented on CASSANDRA-4218:
-------------------------------------------
Also, do you see the words "Compacted data files will not be removed promptly" in your server log anywhere?
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267855#comment-13267855 ]
Jonathan Ellis commented on CASSANDRA-4218:
-------------------------------------------
enabling debug logging on org.apache.cassandra.config.DatabaseDescriptor would show if that last is the problem
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-4218) NPE thrown after
switching to LeveledCompactionStrategy
Posted by "Alex Whittaker (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267871#comment-13267871 ]
Alex Whittaker edited comment on CASSANDRA-4218 at 5/3/12 10:18 PM:
--------------------------------------------------------------------
OK here is the debug output - none seem to go over the limit though there are some very large ones:
[root@babble /opt/werinteractive/cassandra]# grep -E 'DEBUG|ERROR' !$
DEBUG [main] 2012-05-03 22:05:53,938 DatabaseDescriptor.java (line 160) Syncing log with a period of 10000
DEBUG [main] 2012-05-03 22:05:54,162 DatabaseDescriptor.java (line 348) setting auto_bootstrap to true
DEBUG [FlushWriter:1] 2012-05-03 22:06:22,960 DatabaseDescriptor.java (line 792) expected data files size is 733; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,012 DatabaseDescriptor.java (line 792) expected data files size is 733; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,026 DatabaseDescriptor.java (line 792) expected data files size is 838; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,040 DatabaseDescriptor.java (line 792) expected data files size is 834; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,054 DatabaseDescriptor.java (line 792) expected data files size is 4639; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,076 DatabaseDescriptor.java (line 792) expected data files size is 3622; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,092 DatabaseDescriptor.java (line 792) expected data files size is 7340; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,108 DatabaseDescriptor.java (line 792) expected data files size is 2489268; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:7] 2012-05-03 22:06:23,113 DatabaseDescriptor.java (line 792) expected data files size is 1774771; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,474 DatabaseDescriptor.java (line 792) expected data files size is 106525; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:8] 2012-05-03 22:06:23,484 DatabaseDescriptor.java (line 792) expected data files size is 358236723213; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,508 DatabaseDescriptor.java (line 792) expected data files size is 133; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:9] 2012-05-03 22:06:23,509 DatabaseDescriptor.java (line 792) expected data files size is 25959367; largest free partition has 394869746176 bytes free
ERROR [CompactionExecutor:8] 2012-05-03 22:06:23,514 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:8,1,main]
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,522 DatabaseDescriptor.java (line 792) expected data files size is 517; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,566 DatabaseDescriptor.java (line 792) expected data files size is 37; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:12] 2012-05-03 22:06:23,579 DatabaseDescriptor.java (line 792) expected data files size is 1123; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,621 DatabaseDescriptor.java (line 792) expected data files size is 70; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,717 DatabaseDescriptor.java (line 792) expected data files size is 34; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,734 DatabaseDescriptor.java (line 792) expected data files size is 34; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:16] 2012-05-03 22:06:23,746 DatabaseDescriptor.java (line 792) expected data files size is 1192; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:21] 2012-05-03 22:11:10,020 DatabaseDescriptor.java (line 792) expected data files size is 358236723213; largest free partition has 394871987200 bytes free
ERROR [CompactionExecutor:21] 2012-05-03 22:11:10,035 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:21,1,main]
was (Author: prolixative):
OK here is the debug output - none seem to go over the limit though there are some very large ones:
[root@babble /opt/werinteractive/cassandra]# grep DEBUG !$
grep DEBUG /opt/werinteractive/log/cassandra_system.log
DEBUG [main] 2012-05-03 22:05:53,938 DatabaseDescriptor.java (line 160) Syncing log with a period of 10000
DEBUG [main] 2012-05-03 22:05:54,162 DatabaseDescriptor.java (line 348) setting auto_bootstrap to true
DEBUG [FlushWriter:1] 2012-05-03 22:06:22,960 DatabaseDescriptor.java (line 792) expected data files size is 733; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,012 DatabaseDescriptor.java (line 792) expected data files size is 733; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,026 DatabaseDescriptor.java (line 792) expected data files size is 838; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,040 DatabaseDescriptor.java (line 792) expected data files size is 834; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,054 DatabaseDescriptor.java (line 792) expected data files size is 4639; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,076 DatabaseDescriptor.java (line 792) expected data files size is 3622; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,092 DatabaseDescriptor.java (line 792) expected data files size is 7340; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,108 DatabaseDescriptor.java (line 792) expected data files size is 2489268; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:7] 2012-05-03 22:06:23,113 DatabaseDescriptor.java (line 792) expected data files size is 1774771; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,474 DatabaseDescriptor.java (line 792) expected data files size is 106525; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:8] 2012-05-03 22:06:23,484 DatabaseDescriptor.java (line 792) expected data files size is 358236723213; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,508 DatabaseDescriptor.java (line 792) expected data files size is 133; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:9] 2012-05-03 22:06:23,509 DatabaseDescriptor.java (line 792) expected data files size is 25959367; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,522 DatabaseDescriptor.java (line 792) expected data files size is 517; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,566 DatabaseDescriptor.java (line 792) expected data files size is 37; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:12] 2012-05-03 22:06:23,579 DatabaseDescriptor.java (line 792) expected data files size is 1123; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,621 DatabaseDescriptor.java (line 792) expected data files size is 70; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,717 DatabaseDescriptor.java (line 792) expected data files size is 34; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,734 DatabaseDescriptor.java (line 792) expected data files size is 34; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:16] 2012-05-03 22:06:23,746 DatabaseDescriptor.java (line 792) expected data files size is 1192; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:21] 2012-05-03 22:11:10,020 DatabaseDescriptor.java (line 792) expected data files size is 358236723213; largest free partition has 394871987200 bytes free
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267883#comment-13267883 ]
Jonathan Ellis commented on CASSANDRA-4218:
-------------------------------------------
finally: it looks like you're running with assertions disabled, is that correct?
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267846#comment-13267846 ]
Jonathan Ellis commented on CASSANDRA-4218:
-------------------------------------------
Do you have multiple data directories configured?
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Alex Whittaker (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267871#comment-13267871 ]
Alex Whittaker commented on CASSANDRA-4218:
-------------------------------------------
OK here is the debug output - none seem to go over the limit though there are some very large ones:
[root@babble /opt/werinteractive/cassandra]# grep DEBUG !$
grep DEBUG /opt/werinteractive/log/cassandra_system.log
DEBUG [main] 2012-05-03 22:05:53,938 DatabaseDescriptor.java (line 160) Syncing log with a period of 10000
DEBUG [main] 2012-05-03 22:05:54,162 DatabaseDescriptor.java (line 348) setting auto_bootstrap to true
DEBUG [FlushWriter:1] 2012-05-03 22:06:22,960 DatabaseDescriptor.java (line 792) expected data files size is 733; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,012 DatabaseDescriptor.java (line 792) expected data files size is 733; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,026 DatabaseDescriptor.java (line 792) expected data files size is 838; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,040 DatabaseDescriptor.java (line 792) expected data files size is 834; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,054 DatabaseDescriptor.java (line 792) expected data files size is 4639; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,076 DatabaseDescriptor.java (line 792) expected data files size is 3622; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,092 DatabaseDescriptor.java (line 792) expected data files size is 7340; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,108 DatabaseDescriptor.java (line 792) expected data files size is 2489268; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:7] 2012-05-03 22:06:23,113 DatabaseDescriptor.java (line 792) expected data files size is 1774771; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,474 DatabaseDescriptor.java (line 792) expected data files size is 106525; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:8] 2012-05-03 22:06:23,484 DatabaseDescriptor.java (line 792) expected data files size is 358236723213; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,508 DatabaseDescriptor.java (line 792) expected data files size is 133; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:9] 2012-05-03 22:06:23,509 DatabaseDescriptor.java (line 792) expected data files size is 25959367; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,522 DatabaseDescriptor.java (line 792) expected data files size is 517; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,566 DatabaseDescriptor.java (line 792) expected data files size is 37; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:12] 2012-05-03 22:06:23,579 DatabaseDescriptor.java (line 792) expected data files size is 1123; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,621 DatabaseDescriptor.java (line 792) expected data files size is 70; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,717 DatabaseDescriptor.java (line 792) expected data files size is 34; largest free partition has 394869746176 bytes free
DEBUG [FlushWriter:1] 2012-05-03 22:06:23,734 DatabaseDescriptor.java (line 792) expected data files size is 34; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:16] 2012-05-03 22:06:23,746 DatabaseDescriptor.java (line 792) expected data files size is 1192; largest free partition has 394869746176 bytes free
DEBUG [CompactionExecutor:21] 2012-05-03 22:11:10,020 DatabaseDescriptor.java (line 792) expected data files size is 358236723213; largest free partition has 394871987200 bytes free
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267860#comment-13267860 ]
Jonathan Ellis commented on CASSANDRA-4218:
-------------------------------------------
(log4j-server.properties can be edited w/o restarting)
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Alex Whittaker (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267850#comment-13267850 ]
Alex Whittaker commented on CASSANDRA-4218:
-------------------------------------------
No just the single one - a bit more context around the call stack, this from the same thread:
INFO [FlushWriter:1] 2012-05-03 21:34:53,918 Memtable.java (line 246) Writing Memtable
-PlayrUser.PlayrUser_FirstName_Index@913631246(628/785 serialized/live bytes, 14 ops)
INFO [CompactionExecutor:10] 2012-05-03 21:34:53,927 CompactionTask.java (line 115) Co
mpacting [SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/Play
rUserData-hb-29747-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandr
a/data/IAMPLAYR/PlayrUserData-hb-29757-Data.db'), SSTableReader(path='/opt/werinteracti
ve/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29750-Data.db'), SSTableReader(pat
h='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29759-Data.db'
), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserDa
ta-hb-29768-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/
IAMPLAYR/PlayrUserData-hb-29471-Data.db'), SSTableReader(path='/opt/werinteractive/data
base/cassandra/data/IAMPLAYR/PlayrUserData-hb-29758-Data.db'), SSTableReader(path='/opt
/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29728-Data.db'), SSTa
bleReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29762-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29745-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-27034-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29764-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29752-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29751-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29766-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hc-29865-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29774-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29740-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29441-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29324-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29767-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29734-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29760-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29761-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29755-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29754-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29546-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29743-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29776-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29742-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29769-Data.db'), SSTableReader(path='/opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUserData-hb-29775-Data.db')]
INFO [FlushWriter:1] 2012-05-03 21:34:53,928 Memtable.java (line 283) Completed flushing /opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUser.PlayrUser_FirstName_Index-hc-2241-Data.db (1235 bytes)
INFO [FlushWriter:1] 2012-05-03 21:34:53,930 Memtable.java (line 246) Writing Memtable-PlayrUser.PlayrUser_LastPortal_Index@1326770967(734/917 serialized/live bytes, 16 ops)
INFO [FlushWriter:1] 2012-05-03 21:34:53,939 Memtable.java (line 283) Completed flushing /opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUser.PlayrUser_LastPortal_Index-hc-1162-Data.db (762 bytes)
ERROR [CompactionExecutor:10] 2012-05-03 21:34:53,941 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:10,1,main]
java.lang.NullPointerException
at java.io.File.<init>(File.java:222)
at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [FlushWriter:1] 2012-05-03 21:34:53,943 Memtable.java (line 246) Writing Memtable-PlayrUser.PlayrUser_OriginalPortal_Index@1535464089(628/785 serialized/live bytes, 14 ops)
INFO [FlushWriter:1] 2012-05-03 21:34:53,955 Memtable.java (line 283) Completed flushing /opt/werinteractive/database/cassandra/data/IAMPLAYR/PlayrUser.PlayrUser_OriginalPortal_Index-hc-1167-Data.db (608 bytes)
There are two machines in this state, the other four are fine.
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267884#comment-13267884 ]
Jonathan Ellis commented on CASSANDRA-4218:
-------------------------------------------
bq. Are there likely to be any issues if we switched back to size tier compaction?
No.
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267907#comment-13267907 ]
Jonathan Ellis commented on CASSANDRA-4218:
-------------------------------------------
Turning on compression is a no-op on existing sstables, it will only affect newly created ones (flushed or compacted). You can run upgradesstables to re-write existing ones w/ compression.
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Alex Whittaker (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267888#comment-13267888 ]
Alex Whittaker commented on CASSANDRA-4218:
-------------------------------------------
Yes assertions are disabled
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Alex Whittaker (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267863#comment-13267863 ]
Alex Whittaker commented on CASSANDRA-4218:
-------------------------------------------
Are there likely to be any issues if we switched back to size tier compaction?
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-4218.
---------------------------------------
Resolution: Not A Problem
If you don't have enough spare space to re-level all your sstables at once (or Cassandra thinks you don't), you can move some of them off-server, level the rest, then copy the others back and inject them with {{nodetool refresh}} (no restart required). Or, maybe you have some snapshots you can clean out to free up space.
Finally, running without -ea makes troubleshooting harder. :)
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267811#comment-13267811 ]
Pavel Yaskevich commented on CASSANDRA-4218:
--------------------------------------------
It looks like this was fixed by CASSANDRA-3985.
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267853#comment-13267853 ]
Jonathan Ellis commented on CASSANDRA-4218:
-------------------------------------------
One other note: Cassandra goes by File.getUsableSpace, not necessarily total disk space. So if you have quota or other security issues that could affect it.
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Alex Whittaker (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267837#comment-13267837 ]
Alex Whittaker commented on CASSANDRA-4218:
-------------------------------------------
Yes I'm afraid so and there is no shortage of disk space as well I'm afraid.
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267836#comment-13267836 ]
Jonathan Ellis commented on CASSANDRA-4218:
-------------------------------------------
Alex, are you still seeing this in 1.0.9?
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Alex Whittaker (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267902#comment-13267902 ]
Alex Whittaker commented on CASSANDRA-4218:
-------------------------------------------
*sigh* so we switched to levelled because we were almost out of disk space but that was not enough to help us I guess. We've switched back to size tiered now - if we switch on compression will it be prioritised above compaction?
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4218) NPE thrown after switching to
LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267897#comment-13267897 ]
Jonathan Ellis commented on CASSANDRA-4218:
-------------------------------------------
{{noformat}}
DEBUG [CompactionExecutor:21] 2012-05-03 22:11:10,020 DatabaseDescriptor.java (line 792) expected data files size is 358236723213; largest free partition has 394871987200 bytes free
{{noformat}}
Well, that's your smoking gun. The log message is a bit misleading, because of this code:
{{code}}
. logger.debug("expected data files size is {}; largest free partition has {} bytes free",
expectedCompactedFileSize,
maxFreeDisk);
// Load factor of 0.9 we do not want to use the entire disk that is too risky.
maxFreeDisk = (long) (0.9 * maxFreeDisk);
{{code}}
... and sure enough, {{358236723213 < 394871987200 * 0.9}}.
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-4218) NPE thrown after
switching to LeveledCompactionStrategy
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267897#comment-13267897 ]
Jonathan Ellis edited comment on CASSANDRA-4218 at 5/3/12 10:39 PM:
--------------------------------------------------------------------
{noformat}
DEBUG [CompactionExecutor:21] 2012-05-03 22:11:10,020 DatabaseDescriptor.java (line 792) expected data files size is 358236723213; largest free partition has 394871987200 bytes free
{noformat}
Well, that's your smoking gun. The log message is a bit misleading, because of this code:
{code}
. logger.debug("expected data files size is {}; largest free partition has {} bytes free",
expectedCompactedFileSize,
maxFreeDisk);
// Load factor of 0.9 we do not want to use the entire disk that is too risky.
maxFreeDisk = (long) (0.9 * maxFreeDisk);
{code}
... and sure enough, {{358236723213 < 394871987200 * 0.9}}.
was (Author: jbellis):
{{noformat}}
DEBUG [CompactionExecutor:21] 2012-05-03 22:11:10,020 DatabaseDescriptor.java (line 792) expected data files size is 358236723213; largest free partition has 394871987200 bytes free
{{noformat}}
Well, that's your smoking gun. The log message is a bit misleading, because of this code:
{{code}}
. logger.debug("expected data files size is {}; largest free partition has {} bytes free",
expectedCompactedFileSize,
maxFreeDisk);
// Load factor of 0.9 we do not want to use the entire disk that is too risky.
maxFreeDisk = (long) (0.9 * maxFreeDisk);
{{code}}
... and sure enough, {{358236723213 < 394871987200 * 0.9}}.
> NPE thrown after switching to LeveledCompactionStrategy
> -------------------------------------------------------
>
> Key: CASSANDRA-4218
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4218
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5, 1.0.9
> Environment: SunOS corbina.local 5.11 joyent_ i86pc i386 i86pc Solaris
> Reporter: Alex Whittaker
>
> After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:
> ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
> java.lang.NullPointerException
> at java.io.File.<init>(File.java:222)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
> at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira