You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sylvain Lebresne (Created) (JIRA)" <ji...@apache.org> on 2012/01/11 10:40:39 UTC
[jira] [Created] (CASSANDRA-3727) Fix unit tests failure
Fix unit tests failure
----------------------
Key: CASSANDRA-3727
URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
Project: Cassandra
Issue Type: Bug
Components: Tests
Affects Versions: 1.0.7
Reporter: Sylvain Lebresne
Priority: Blocker
Fix For: 1.0.7
On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
* CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
* SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
* RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
* I also see a bunch of stack trace with errors like:
{noformat}
[junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
[junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
{noformat}
(with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Mark Butler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189139#comment-13189139 ]
Mark Butler commented on CASSANDRA-3727:
----------------------------------------
Building Cassandra 1.0.7 I get one test failure:
{code}
Class org.apache.cassandra.db.compaction.CompactionsTest
Name Tests Errors Failures Time(s) Time Stamp Host
CompactionsTest 1 1 0 0.000 2012-01-19T12:52:18 mbutler-OptiPlex-990
Tests
Name Status Type Time(s)
testSuperColumnCompactions Error Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
0.001
{code}
and also see exceptions being thrown saying it was unable to create a hard link
{code}
[junit] WARN 13:53:33,357 Overriding RING_DELAY to 1000ms
[junit] ERROR 13:53:39,794 Unable to create hard link
[junit] com.sun.jna.LastErrorException: errno was 17
[junit] at org.apache.cassandra.utils.CLibrary.link(Native Method)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:146)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] ERROR 13:53:39,796 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
[junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Standard1-hc-1-Digest.sha1 to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Standard1-hc-1-Digest.sha1 (errno 17)
[junit] at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] Caused by: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Standard1-hc-1-Digest.sha1 to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Standard1-hc-1-Digest.sha1 (errno 17)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:160)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] ... 8 more
[junit] ERROR 13:53:39,797 Unable to create hard link
[junit] com.sun.jna.LastErrorException: errno was 17
[junit] at org.apache.cassandra.utils.CLibrary.link(Native Method)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:146)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] ERROR 13:53:39,798 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
[junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Standard1-hc-2-Filter.db to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Standard1-hc-2-Filter.db (errno 17)
[junit] at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] Caused by: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Standard1-hc-2-Filter.db to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Standard1-hc-2-Filter.db (errno 17)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:160)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] ... 8 more
[junit] ERROR 13:53:41,501 Unable to create hard link
[junit] com.sun.jna.LastErrorException: errno was 17
[junit] at org.apache.cassandra.utils.CLibrary.link(Native Method)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:146)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] ERROR 13:53:41,502 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
[junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1.626972746864617465-hc-1-Filter.db to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Indexed1.626972746864617465-hc-1-Filter.db (errno 17)
[junit] at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] Caused by: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1.626972746864617465-hc-1-Filter.db to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Indexed1.626972746864617465-hc-1-Filter.db (errno 17)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:160)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] ... 8 more
[junit] ERROR 13:53:41,504 Unable to create hard link
[junit] com.sun.jna.LastErrorException: errno was 17
[junit] at org.apache.cassandra.utils.CLibrary.link(Native Method)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:146)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] ERROR 13:53:41,505 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
[junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Digest.sha1 to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Digest.sha1 (errno 17)
[junit] at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] Caused by: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Digest.sha1 to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Digest.sha1 (errno 17)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:160)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] ... 8 more
[junit] ERROR 13:53:41,802 Unable to create hard link
[junit] com.sun.jna.LastErrorException: errno was 17
[junit] at org.apache.cassandra.utils.CLibrary.link(Native Method)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:146)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] ERROR 13:53:41,802 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
[junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1.626972746864617465-hc-2-Digest.sha1 to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Indexed1.626972746864617465-hc-2-Digest.sha1 (errno 17)
[junit] at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] Caused by: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1.626972746864617465-hc-2-Digest.sha1 to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Indexed1.626972746864617465-hc-2-Digest.sha1 (errno 17)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:160)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] ... 8 more
[junit] ERROR 13:53:41,804 Unable to create hard link
[junit] com.sun.jna.LastErrorException: errno was 17
[junit] at org.apache.cassandra.utils.CLibrary.link(Native Method)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:146)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] ERROR 13:53:41,805 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
[junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Filter.db to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Filter.db (errno 17)
[junit] at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] Caused by: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Filter.db to /home/mbutler/workspace/vscc/vscc-oss-components/cassandra-1.0.7/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Filter.db (errno 17)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:160)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] ... 8 more
[junit] ------------- ---------------- ---------------
{code}
Any suggestions on a fix?
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183955#comment-13183955 ]
Pavel Yaskevich commented on CASSANDRA-3727:
--------------------------------------------
CliTest and others should be timeouting because of newly added shutdown hook.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184335#comment-13184335 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
bq. RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently).
FWIW, I get a timeout instead (on Windows).
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184361#comment-13184361 ]
Brandon Williams commented on CASSANDRA-3727:
---------------------------------------------
Same 'in use by another process' under linux. There is definitely no other process.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184681#comment-13184681 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
User error... it was my paused Cassandra instance in my IDE holding that CL segment open. With that figured out, I also pushed 1e750138177e9cd9cbd6537451a4b5cd301dab3a which allows MS to be restarted by RemoveTest.
All the tests now pass for me, except RecoveryManagerTruncateTest which has failed on Windows for 0.8+.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3727:
--------------------------------------
Attachment: 3727.txt
So, I was over-optimistic in CASSANDRA-3335 when I thought I could get by without a MessagingService shutdown method. The problem is that although my changes there do work to prevent accepting new connections, and to stop work on existing connections *after the first command post-shutdown*, that's not good enough in this case since the client is just sitting on its connection and never sends another command.
So, this patch renames MS.waitForCallbacks() back to shutdown(), and refuses to add new callbacks after that. However, the analysis on 3335 that there's no good way to deal with an exception here, so we do this instead in ExpiringMap:
{code}
. public V put(K key, V value, long timeout)
{
if (shutdown)
{
// StorageProxy isn't equipped to deal with "I'm nominally alive, but I can't send any messages out."
// So we'll just sit on this thread for until the rest of the server shutdown completes.
//
// See comments in CustomTThreadPoolServer.serve, CASSANDRA-3335, and CASSANDRA-3727.
try
{
Thread.sleep(Long.MAX_VALUE);
}
catch (InterruptedException e)
{
throw new AssertionError(e);
}
}
CacheableObject<V> previous = cache.put(key, new CacheableObject<V>(value, timeout));
return (previous == null) ? null : previous.getValue();
}
{code}
Then, we switch the Thrift executor (and all DTPE instances) to use daemon threads, and remove the wait-for-WorkerProcess threads code from CustomTThreadPoolServer.serve:
{code}
. // Thrift's default shutdown waits for the WorkerProcess threads to complete. We do not,
// because doing that allows a client to hold our shutdown "hostage" by simply not sending
// another message after stop is called (since process will block indefinitely trying to read
// the next meessage header).
//
// The "right" fix would be to update thrift to set a socket timeout on client connections
// (and tolerate unintentional timeouts until stopped_ is set). But this requires deep
// changes to the code generator, so simply setting these threads to daemon (in our custom
// CleaningThreadPool) and ignoring them after shutdown is good enough.
//
// Remember, our goal on shutdown is not necessarily that each client request we receive
// gets answered first [to do that, you should redirect clients to a different coordinator
// first], but rather (1) to make sure that for each update we ack as successful, we generate
// hints for any non-responsive replicas, and (2) to make sure that we quickly stop
// accepting client connections so shutdown can continue. Not waiting for the WorkerProcess
// threads here accomplishes (2); MessagingService's shutdown method takes care of (1).
//
// See CASSANDRA-3335 and CASSANDRA-3727.
{code}
Finally, this patch also updates Memtable's memorymeter thread to use the newly daemonized DTPE for good measure, since there's no reason to ever block shutdown for that either.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189145#comment-13189145 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
See CASSANDRA-3735 for the hard link fix
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183957#comment-13183957 ]
Sylvain Lebresne commented on CASSANDRA-3727:
---------------------------------------------
Are you saying that they should be timeouting as in 'having them timeouting is a feature' or are you just pointing out the likely source of the problem? In the latter, do you remember the ticket that introduced those (or better, have a fix for it)?
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184328#comment-13184328 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
SystemTableTest failure taken care of on CASSANDRA-3579
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183960#comment-13183960 ]
Pavel Yaskevich commented on CASSANDRA-3727:
--------------------------------------------
I'm saying that it's likely source of the problem and for CliTest fix would be pretty straightforward, make CliMain to disconnect properly after tests are done (I'm going to attach a patch fixing patch in a few minutes to this ticket), I don't know about other tests tho.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184495#comment-13184495 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
Pushed 452ddf63c530fc573551e6fc9c79c1a876f11dd0 with the socket teardown code added back. Now it's hitting {{java.io.IOException: Failed to delete c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\commitlog\CommitLog-1326319330071.log}}. Progress?
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184166#comment-13184166 ]
Pavel Yaskevich commented on CASSANDRA-3727:
--------------------------------------------
It's easy to test - run cassandra and in other terminal session connect to it using CLI and try Ctrl-C Cassandra server without closing CLI.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184249#comment-13184249 ]
Brandon Williams commented on CASSANDRA-3727:
---------------------------------------------
+1
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184091#comment-13184091 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
Hmm. Can we fix the thrift shutdown instead to not wait for sockets to be closed nicely?
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184158#comment-13184158 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
Definitely not, otherwise there is no way to shut down if clients stay connected
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184273#comment-13184273 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
committed; leaving open for other test failures
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184340#comment-13184340 ]
Pavel Yaskevich commented on CASSANDRA-3727:
--------------------------------------------
I get the same thing Sylvain does in RemoveTest : '/127.0.0.1:7010 is in use by another process' on Mac OS X.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184163#comment-13184163 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
bq. Can we fix the thrift shutdown instead to not wait for sockets to be closed nicely
I'm not sure why this isn't how it already works. From CustomTThreadPoolServer.WorkerProcess:
{code}
. while (!stopped_ && processor.process(inputProtocol, outputProtocol))
{
inputProtocol = inputProtocolFactory_.getProtocol(inputTransport);
outputProtocol = outputProtocolFactory_.getProtocol(outputTransport);
}
{code}
In other words, as soon as stopped_ is set (by the stop() method), each thread should finish the current request but not accept more.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184094#comment-13184094 ]
Pavel Yaskevich commented on CASSANDRA-3727:
--------------------------------------------
I guess we can, but I thought that it's kind of feature that it does wait...
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3727) Fix unit tests failure
Posted by "Sylvain Lebresne (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne resolved CASSANDRA-3727.
-----------------------------------------
Resolution: Fixed
I still had intermittent failure of columnFamilyStoreTest, but because SystemTable.isIndexBuilt() was suffering from the same 'I forgot to expunge tombstones' problem than SystemTable.loadTokens(). I took on myself to commit the same fix for that instance directly (and check no other method had this problem).
So closing this as all tests are now passing.
However I'd be interested to know if anyone else is seeing the 'unable to create link' stack trace during tests, because if so we should probably open another ticket to investigate.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184436#comment-13184436 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
RemoveTest is trying to stop/start MessagingService multiple times in the same suite (see setup/tearDown methods). My guess is that worked well enough pre-CASSANDRA-3335. I'll see about making it happy again, although this feels fragile.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: 3727.txt, CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3727) Fix unit tests failure
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184159#comment-13184159 ]
Jonathan Ellis commented on CASSANDRA-3727:
-------------------------------------------
(The thrift shutdown was introduced for CASSANDRA-3335)
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3727) Fix unit tests failure
Posted by "Pavel Yaskevich (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-3727:
---------------------------------------
Attachment: CASSANDRA-3727-CliTest-timeout-fix.patch
fixed CliTest timeout by correctly closing transport connection which allows Cassandra shutdown hook to proceed without waiting for RPC connections to close.
> Fix unit tests failure
> ----------------------
>
> Key: CASSANDRA-3727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3727
> Project: Cassandra
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.7
> Reporter: Sylvain Lebresne
> Priority: Blocker
> Fix For: 1.0.7
>
> Attachments: CASSANDRA-3727-CliTest-timeout-fix.patch
>
>
> On current 1.0 branch (and on my machine: Linux), I have the following unit test failures:
> * CliTest and EmbeddedCassandraTest: they both first kind of pass (JUnit first prints a message with no failures in it), then hang until JUnit timeout and fails with a 'Timeout occurred'. In other word, the tests themselves are passing, but something they do prevents the process to exit cleanly leading to a JUnit timeout. I don't want to discard that as not a problem, because if something can make the process not exit cleanly, this can be a pain for restarts (and in particular upgrades) and hence would be basically a regression. I'm marking the ticket as blocker (for the release of 1.0.7) mostly because of this one.
> * SystemTableTest: throws an assertionError. I haven't checked yet, so that could be an easy one to fix.
> * RemoveTest: it fails, saying that '/127.0.0.1:7010 is in use by another process' (consistently). But I have no other process running on port 7010. It's likely just of problem of the test, but it's new and in the meantime removes are not tested.
> * I also see a bunch of stack trace with errors like:
> {noformat}
> [junit] ERROR 10:01:59,007 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> [junit] java.lang.RuntimeException: java.io.IOException: Unable to create hard link from build/test/cassandra/data/Keyspace1/Indexed1-hc-1-Index.db to /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups/Indexed1-hc-1-Index.db (errno 17)
> {noformat}
> (with SSTableReaderTest). This does not make the tests fail, but it is still worth investigating. It may be due to CASSANDRA-3101.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira