You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2009/08/27 22:46:59 UTC
[jira] Created: (CASSANDRA-395) BootstrapTest occasionally fails
BootstrapTest occasionally fails
--------------------------------
Key: CASSANDRA-395
URL: https://issues.apache.org/jira/browse/CASSANDRA-395
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Jonathan Ellis
Assignee: Sandeep Tata
sample failure:
[junit] Testcase: testAntiCompaction1(org.apache.cassandra.db.BootstrapTest): Caused an ERROR
[junit] /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
[junit] java.io.FileNotFoundException: /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
[junit] at java.io.RandomAccessFile.open(Native Method)
[junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
[junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
[junit] at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:142)
[junit] at org.apache.cassandra.io.FileStruct.<init>(FileStruct.java:49)
[junit] at org.apache.cassandra.io.SSTableReader.getFileStruct(SSTableReader.java:321)
[junit] at org.apache.cassandra.db.ColumnFamilyStore.initializePriorityQueue(ColumnFamilyStore.java:655)
[junit] at org.apache.cassandra.db.ColumnFamilyStore.doFileAntiCompaction(ColumnFamilyStore.java:911)
[junit] at org.apache.cassandra.db.ColumnFamilyStore.doAntiCompaction(ColumnFamilyStore.java:814)
[junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction(BootstrapTest.java:63)
[junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction1(BootstrapTest.java:72)
I have seen someone else mention this on IRC too. Most of the time, the test passes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-395) BootstrapTest occasionally fails
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748846#action_12748846 ]
Hudson commented on CASSANDRA-395:
----------------------------------
Integrated in Cassandra #180 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/180/])
call doAntiCompaction in the tests the same way we do in the actual code -- by submitting it to the MinorCompactionManager so it serializes with other compaction tasks. patch by Sandeep Tata; reviewed by jbellis for
> BootstrapTest occasionally fails
> --------------------------------
>
> Key: CASSANDRA-395
> URL: https://issues.apache.org/jira/browse/CASSANDRA-395
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Sandeep Tata
> Fix For: 0.4
>
> Attachments: 395.patch
>
>
> sample failure:
> [junit] Testcase: testAntiCompaction1(org.apache.cassandra.db.BootstrapTest): Caused an ERROR
> [junit] /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
> [junit] java.io.FileNotFoundException: /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
> [junit] at java.io.RandomAccessFile.open(Native Method)
> [junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> [junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> [junit] at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:142)
> [junit] at org.apache.cassandra.io.FileStruct.<init>(FileStruct.java:49)
> [junit] at org.apache.cassandra.io.SSTableReader.getFileStruct(SSTableReader.java:321)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.initializePriorityQueue(ColumnFamilyStore.java:655)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.doFileAntiCompaction(ColumnFamilyStore.java:911)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.doAntiCompaction(ColumnFamilyStore.java:814)
> [junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction(BootstrapTest.java:63)
> [junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction1(BootstrapTest.java:72)
> I have seen someone else mention this on IRC too. Most of the time, the test passes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-395) BootstrapTest occasionally fails
Posted by "Sandeep Tata (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748629#action_12748629 ]
Sandeep Tata commented on CASSANDRA-395:
----------------------------------------
Quick Fix: Change MemtableObjectCountInMillions to 0.0002. This allows memtables to be bigger and avoids kicking off a compaction before the anticompaction can complete. BootstrapTest will no longer exhibit intermittent failures. (verified)
But that's just a band-aid -- the real solution is to synchronize compaction & anticompaction correctly.
> BootstrapTest occasionally fails
> --------------------------------
>
> Key: CASSANDRA-395
> URL: https://issues.apache.org/jira/browse/CASSANDRA-395
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Sandeep Tata
> Fix For: 0.4
>
>
> sample failure:
> [junit] Testcase: testAntiCompaction1(org.apache.cassandra.db.BootstrapTest): Caused an ERROR
> [junit] /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
> [junit] java.io.FileNotFoundException: /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
> [junit] at java.io.RandomAccessFile.open(Native Method)
> [junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> [junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> [junit] at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:142)
> [junit] at org.apache.cassandra.io.FileStruct.<init>(FileStruct.java:49)
> [junit] at org.apache.cassandra.io.SSTableReader.getFileStruct(SSTableReader.java:321)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.initializePriorityQueue(ColumnFamilyStore.java:655)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.doFileAntiCompaction(ColumnFamilyStore.java:911)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.doAntiCompaction(ColumnFamilyStore.java:814)
> [junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction(BootstrapTest.java:63)
> [junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction1(BootstrapTest.java:72)
> I have seen someone else mention this on IRC too. Most of the time, the test passes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-395) BootstrapTest occasionally fails
Posted by "Sandeep Tata (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sandeep Tata updated CASSANDRA-395:
-----------------------------------
Attachment: 395.patch
Patch to call doAntiCompaction in the tests the same way we do in the actual code -- by submitting it to the MinorCompactionManager so it serializes with other compaction tasks.
Ran the tests a 200 times -- I don't see failures anymore.
> BootstrapTest occasionally fails
> --------------------------------
>
> Key: CASSANDRA-395
> URL: https://issues.apache.org/jira/browse/CASSANDRA-395
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Sandeep Tata
> Fix For: 0.4
>
> Attachments: 395.patch
>
>
> sample failure:
> [junit] Testcase: testAntiCompaction1(org.apache.cassandra.db.BootstrapTest): Caused an ERROR
> [junit] /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
> [junit] java.io.FileNotFoundException: /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
> [junit] at java.io.RandomAccessFile.open(Native Method)
> [junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> [junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> [junit] at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:142)
> [junit] at org.apache.cassandra.io.FileStruct.<init>(FileStruct.java:49)
> [junit] at org.apache.cassandra.io.SSTableReader.getFileStruct(SSTableReader.java:321)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.initializePriorityQueue(ColumnFamilyStore.java:655)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.doFileAntiCompaction(ColumnFamilyStore.java:911)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.doAntiCompaction(ColumnFamilyStore.java:814)
> [junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction(BootstrapTest.java:63)
> [junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction1(BootstrapTest.java:72)
> I have seen someone else mention this on IRC too. Most of the time, the test passes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-395) BootstrapTest occasionally fails
Posted by "Sandeep Tata (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748627#action_12748627 ]
Sandeep Tata commented on CASSANDRA-395:
----------------------------------------
Ah this is a fun bug -- looks like this happens if the following sequence occurs:
1. Call to flush memtable
2. Compaction submitted (from storeLocation after flushing memtable) but not yet completed.
3. CFS.doAntiCompaction reads keys from ssTables_
4. Compaction completes
5. SSTableReader.getApproximateKeyCount(files) now points to files that don't exist!
We have to prevent compaction from messing up the list of files that anticompaction needs to work on to build the new file.
I didn't expect this would happen in a short Junit test. Need to see why compaction is getting triggered.
> BootstrapTest occasionally fails
> --------------------------------
>
> Key: CASSANDRA-395
> URL: https://issues.apache.org/jira/browse/CASSANDRA-395
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Sandeep Tata
> Fix For: 0.4
>
>
> sample failure:
> [junit] Testcase: testAntiCompaction1(org.apache.cassandra.db.BootstrapTest): Caused an ERROR
> [junit] /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
> [junit] java.io.FileNotFoundException: /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
> [junit] at java.io.RandomAccessFile.open(Native Method)
> [junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> [junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> [junit] at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:142)
> [junit] at org.apache.cassandra.io.FileStruct.<init>(FileStruct.java:49)
> [junit] at org.apache.cassandra.io.SSTableReader.getFileStruct(SSTableReader.java:321)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.initializePriorityQueue(ColumnFamilyStore.java:655)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.doFileAntiCompaction(ColumnFamilyStore.java:911)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.doAntiCompaction(ColumnFamilyStore.java:814)
> [junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction(BootstrapTest.java:63)
> [junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction1(BootstrapTest.java:72)
> I have seen someone else mention this on IRC too. Most of the time, the test passes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-395) BootstrapTest occasionally fails
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-395:
-------------------------------------
Fix Version/s: 0.4
> BootstrapTest occasionally fails
> --------------------------------
>
> Key: CASSANDRA-395
> URL: https://issues.apache.org/jira/browse/CASSANDRA-395
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Sandeep Tata
> Fix For: 0.4
>
>
> sample failure:
> [junit] Testcase: testAntiCompaction1(org.apache.cassandra.db.BootstrapTest): Caused an ERROR
> [junit] /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
> [junit] java.io.FileNotFoundException: /home/jonathan/projects/cassandra/git-trunk/build/test/cassandra/data/Keyspace1/Standard1-2-Data.db (No such file or directory)
> [junit] at java.io.RandomAccessFile.open(Native Method)
> [junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> [junit] at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> [junit] at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:142)
> [junit] at org.apache.cassandra.io.FileStruct.<init>(FileStruct.java:49)
> [junit] at org.apache.cassandra.io.SSTableReader.getFileStruct(SSTableReader.java:321)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.initializePriorityQueue(ColumnFamilyStore.java:655)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.doFileAntiCompaction(ColumnFamilyStore.java:911)
> [junit] at org.apache.cassandra.db.ColumnFamilyStore.doAntiCompaction(ColumnFamilyStore.java:814)
> [junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction(BootstrapTest.java:63)
> [junit] at org.apache.cassandra.db.BootstrapTest.testAntiCompaction1(BootstrapTest.java:72)
> I have seen someone else mention this on IRC too. Most of the time, the test passes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.