You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (Created) (JIRA)" <ji...@apache.org> on 2012/01/13 00:11:39 UTC
[jira] [Created] (CASSANDRA-3735) Fix "Unable to create hard link"
SSTableReaderTest error messages
Fix "Unable to create hard link" SSTableReaderTest error messages
-----------------------------------------------------------------
Key: CASSANDRA-3735
URL: https://issues.apache.org/jira/browse/CASSANDRA-3735
Project: Cassandra
Issue Type: Bug
Reporter: Jonathan Ellis
Priority: Minor
Sample failure (on Windows):
{noformat}
[junit] java.io.IOException: Exception while executing the command: cmd /c mklink /H C:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Index.db c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\Standard1-hc-1-Index.db,command error Code: 1, command output: Cannot create a file when that file already exists.
[junit]
[junit] at org.apache.cassandra.utils.CLibrary.exec(CLibrary.java:213)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:188)
[junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:151)
[junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] ERROR 17:10:17,111 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3735) Fix "Unable to
create hard link" SSTableReaderTest error messages
Posted by "Jonathan Ellis (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185330#comment-13185330 ]
Jonathan Ellis edited comment on CASSANDRA-3735 at 1/12/12 11:29 PM:
---------------------------------------------------------------------
Two parts:
first makes it so loadNewSSTables won't reset sstable generation to a lower value than it currently is.
second removes the (hard linked) sstables from the incremental backups directory, so that when loadNewSSTables links them back in they won't conflict.
patch 2 doesn't actually work on Windows because something is keeping the Data component open. Java6 just says "delete failed;" java7's Files.delete says "java.nio.file.FileSystemException: build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Data.db: The process cannot access the file because it is being used by another process." (Even after I prototyped CASSANDRA-3734 to use Files.createLink instead of ProcessBuilder, so that's not the "other process.")
I'm guessing it's counting something (what?) that opened the original Data, towards the link entry as well. But, this should be enough to make the messages go away on Linux, which takes a more permissive view of deleting files opened elsewhere.
was (Author: jbellis):
Two parts:
first makes it so loadNewSSTables won't reset sstable generation to a lower value than it currently is.
second removes the (hard linked) sstables from the incremental backups directory, so that when loadNewSSTables links them back in they won't conflict.
patch 2 doesn't actually work on Windows because something is keeping the Data component open. Java6 just says "delete failed;" java7's Files.delete says "java.nio.file.FileSystemException: build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Data.db: The process cannot access the file because it is being used by another process."
I'm guessing it's counting something (what?) that opened the original Data, towards the link entry as well. But, this should be enough to make the messages go away on Linux, which takes a more permissive view of deleting files opened elsewhere.
> Fix "Unable to create hard link" SSTableReaderTest error messages
> -----------------------------------------------------------------
>
> Key: CASSANDRA-3735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3735
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Attachments: 0001-fix-generation-update-in-loadNewSSTables.patch, 0002-remove-incremental-backups-before-reloading-sstables.patch
>
>
> Sample failure (on Windows):
> {noformat}
> [junit] java.io.IOException: Exception while executing the command: cmd /c mklink /H C:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Index.db c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\Standard1-hc-1-Index.db,command error Code: 1, command output: Cannot create a file when that file already exists.
> [junit]
> [junit] at org.apache.cassandra.utils.CLibrary.exec(CLibrary.java:213)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:188)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:151)
> [junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
> [junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
> [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [junit] at java.lang.Thread.run(Thread.java:662)
> [junit] ERROR 17:10:17,111 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3735) Fix "Unable to create hard link"
SSTableReaderTest error messages
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3735:
--------------------------------------
Attachment: 0003-reset-file-index-generator-on-reset.patch
> Fix "Unable to create hard link" SSTableReaderTest error messages
> -----------------------------------------------------------------
>
> Key: CASSANDRA-3735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3735
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Attachments: 0001-fix-generation-update-in-loadNewSSTables.patch, 0002-remove-incremental-backups-before-reloading-sstables-v2.patch, 0002-remove-incremental-backups-before-reloading-sstables.patch, 0003-reset-file-index-generator-on-reset.patch
>
>
> Sample failure (on Windows):
> {noformat}
> [junit] java.io.IOException: Exception while executing the command: cmd /c mklink /H C:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Index.db c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\Standard1-hc-1-Index.db,command error Code: 1, command output: Cannot create a file when that file already exists.
> [junit]
> [junit] at org.apache.cassandra.utils.CLibrary.exec(CLibrary.java:213)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:188)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:151)
> [junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
> [junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
> [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [junit] at java.lang.Thread.run(Thread.java:662)
> [junit] ERROR 17:10:17,111 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3735) Fix "Unable to create hard link"
SSTableReaderTest error messages
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3735:
--------------------------------------
Attachment: 0002-remove-incremental-backups-before-reloading-sstables-v2.patch
Attaching new version of 0002* that works post CASSANDRA_2794.
> Fix "Unable to create hard link" SSTableReaderTest error messages
> -----------------------------------------------------------------
>
> Key: CASSANDRA-3735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3735
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Attachments: 0001-fix-generation-update-in-loadNewSSTables.patch, 0002-remove-incremental-backups-before-reloading-sstables-v2.patch, 0002-remove-incremental-backups-before-reloading-sstables.patch
>
>
> Sample failure (on Windows):
> {noformat}
> [junit] java.io.IOException: Exception while executing the command: cmd /c mklink /H C:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Index.db c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\Standard1-hc-1-Index.db,command error Code: 1, command output: Cannot create a file when that file already exists.
> [junit]
> [junit] at org.apache.cassandra.utils.CLibrary.exec(CLibrary.java:213)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:188)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:151)
> [junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
> [junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
> [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [junit] at java.lang.Thread.run(Thread.java:662)
> [junit] ERROR 17:10:17,111 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3735) Fix "Unable to
create hard link" SSTableReaderTest error messages
Posted by "Peter Schuller (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200642#comment-13200642 ]
Peter Schuller edited comment on CASSANDRA-3735 at 2/5/12 2:53 AM:
-------------------------------------------------------------------
Attaching new version of 0002* that works post CASSANDRA-2794.
was (Author: scode):
Attaching new version of 0002* that works post CASSANDRA_2794.
> Fix "Unable to create hard link" SSTableReaderTest error messages
> -----------------------------------------------------------------
>
> Key: CASSANDRA-3735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3735
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Attachments: 0001-fix-generation-update-in-loadNewSSTables.patch, 0002-remove-incremental-backups-before-reloading-sstables-v2.patch, 0002-remove-incremental-backups-before-reloading-sstables.patch
>
>
> Sample failure (on Windows):
> {noformat}
> [junit] java.io.IOException: Exception while executing the command: cmd /c mklink /H C:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Index.db c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\Standard1-hc-1-Index.db,command error Code: 1, command output: Cannot create a file when that file already exists.
> [junit]
> [junit] at org.apache.cassandra.utils.CLibrary.exec(CLibrary.java:213)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:188)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:151)
> [junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
> [junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
> [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [junit] at java.lang.Thread.run(Thread.java:662)
> [junit] ERROR 17:10:17,111 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3735) Fix "Unable to create hard
link" SSTableReaderTest error messages
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200666#comment-13200666 ]
Peter Schuller commented on CASSANDRA-3735:
-------------------------------------------
Correction, the conversion to post-2794 did remove the failure of the SSTableReaderTest. Or at least it's no longer happening for me on trunk with the attached patch (v2). Likely because it's only removing specific sstable components given by the iterator, rather than trying to recursively delete backups, but I don't pretend to understand exactly what the history of changes is that caused it to start failing to delete it to begin with.
With repsect to the 'Largest generation seen...' warning, I get that too, but I don't see any subsequent hard link creation failures, nor do I understand why I would if the files are created without using the counter and the quick fix just suppresses the warning? But I'm probably missing something. I do have failing hard linking in ThriftValidationTest though. Maybe this is a side-effect that you're referring to?
In any case, attaching the trivial (if I understood the suggestion correctly) reset patch that supresses the warning.
> Fix "Unable to create hard link" SSTableReaderTest error messages
> -----------------------------------------------------------------
>
> Key: CASSANDRA-3735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3735
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Attachments: 0001-fix-generation-update-in-loadNewSSTables.patch, 0002-remove-incremental-backups-before-reloading-sstables-v2.patch, 0002-remove-incremental-backups-before-reloading-sstables.patch, 0003-reset-file-index-generator-on-reset.patch
>
>
> Sample failure (on Windows):
> {noformat}
> [junit] java.io.IOException: Exception while executing the command: cmd /c mklink /H C:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Index.db c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\Standard1-hc-1-Index.db,command error Code: 1, command output: Cannot create a file when that file already exists.
> [junit]
> [junit] at org.apache.cassandra.utils.CLibrary.exec(CLibrary.java:213)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:188)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:151)
> [junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
> [junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
> [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [junit] at java.lang.Thread.run(Thread.java:662)
> [junit] ERROR 17:10:17,111 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3735) Fix "Unable to
create hard link" SSTableReaderTest error messages
Posted by "Peter Schuller (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200642#comment-13200642 ]
Peter Schuller edited comment on CASSANDRA-3735 at 2/5/12 2:54 AM:
-------------------------------------------------------------------
Attaching new version of 0002* that works (but still with the left-overs already mentioned by jbellis/sylvain) post CASSANDRA-2794.
was (Author: scode):
Attaching new version of 0002* that works post CASSANDRA-2794.
> Fix "Unable to create hard link" SSTableReaderTest error messages
> -----------------------------------------------------------------
>
> Key: CASSANDRA-3735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3735
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Attachments: 0001-fix-generation-update-in-loadNewSSTables.patch, 0002-remove-incremental-backups-before-reloading-sstables-v2.patch, 0002-remove-incremental-backups-before-reloading-sstables.patch
>
>
> Sample failure (on Windows):
> {noformat}
> [junit] java.io.IOException: Exception while executing the command: cmd /c mklink /H C:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Index.db c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\Standard1-hc-1-Index.db,command error Code: 1, command output: Cannot create a file when that file already exists.
> [junit]
> [junit] at org.apache.cassandra.utils.CLibrary.exec(CLibrary.java:213)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:188)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:151)
> [junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
> [junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
> [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [junit] at java.lang.Thread.run(Thread.java:662)
> [junit] ERROR 17:10:17,111 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3735) Fix "Unable to create hard link"
SSTableReaderTest error messages
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3735:
--------------------------------------
Attachment: 0002-remove-incremental-backups-before-reloading-sstables.patch
0001-fix-generation-update-in-loadNewSSTables.patch
Two parts:
first makes it so loadNewSSTables won't reset sstable generation to a lower value than it currently is.
second removes the (hard linked) sstables from the incremental backups directory, so that when loadNewSSTables links them back in they won't conflict.
patch 2 doesn't actually work on Windows because something is keeping the Data component open. Java6 just says "delete failed;" java7's Files.delete says "java.nio.file.FileSystemException: build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Data.db: The process cannot access the file because it is being used by another process."
I'm guessing it's counting something (what?) that opened the original Data, towards the link entry as well. But, this should be enough to make the messages go away on Linux, which takes a more permissive view of deleting files opened elsewhere.
> Fix "Unable to create hard link" SSTableReaderTest error messages
> -----------------------------------------------------------------
>
> Key: CASSANDRA-3735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3735
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Priority: Minor
> Attachments: 0001-fix-generation-update-in-loadNewSSTables.patch, 0002-remove-incremental-backups-before-reloading-sstables.patch
>
>
> Sample failure (on Windows):
> {noformat}
> [junit] java.io.IOException: Exception while executing the command: cmd /c mklink /H C:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Index.db c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\Standard1-hc-1-Index.db,command error Code: 1, command output: Cannot create a file when that file already exists.
> [junit]
> [junit] at org.apache.cassandra.utils.CLibrary.exec(CLibrary.java:213)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:188)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:151)
> [junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
> [junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
> [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [junit] at java.lang.Thread.run(Thread.java:662)
> [junit] ERROR 17:10:17,111 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3735) Fix "Unable to create hard
link" SSTableReaderTest error messages
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190984#comment-13190984 ]
Sylvain Lebresne commented on CASSANDRA-3735:
---------------------------------------------
This seems to handle most of the warnings. However (on 1.0 branch):
* SSTableReaderTest doesn't pass on linux either. It ends up with the following trace:
{noformat}
[junit] Testcase: testPersistentStatisticsFromOlderIndexedSSTable(org.apache.cassandra.io.sstable.SSTableReaderTest): Caused an ERROR
[junit] Failed to delete /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups
[junit] java.io.IOException: Failed to delete /home/mcmanus/Git/cassandra/build/test/cassandra/data/Keyspace1/backups
[junit] at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54)
[junit] at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:237)
[junit] at org.apache.cassandra.io.sstable.SSTableReaderTest.clearAndLoad(SSTableReaderTest.java:167)
[junit] at org.apache.cassandra.io.sstable.SSTableReaderTest.assertIndexQueryWorks(SSTableReaderTest.java:260)
[junit] at org.apache.cassandra.io.sstable.SSTableReaderTest.testPersistentStatisticsFromOlderIndexedSSTable(SSTableReaderTest.java:251)
{noformat}
* I get a "Largest generation seen in loaded sstables was 8, which may overlap with native sstable files (generation 8)" warning during ColumnFamilyStoreTest, followed by a (not surpising) 'Unable to create hard link', in testSliceByNamesCommandOldMetatada. I believe a quick fix could be to have clearUnsafe reset the generation for the CF. Another solution would be to make loadNewSSTables smarter/less fragile by having it 'reserve' a generation for each of the new file to load from the current generation and rename the loaded files accordingly.
> Fix "Unable to create hard link" SSTableReaderTest error messages
> -----------------------------------------------------------------
>
> Key: CASSANDRA-3735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3735
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Attachments: 0001-fix-generation-update-in-loadNewSSTables.patch, 0002-remove-incremental-backups-before-reloading-sstables.patch
>
>
> Sample failure (on Windows):
> {noformat}
> [junit] java.io.IOException: Exception while executing the command: cmd /c mklink /H C:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\backups\Standard1-hc-1-Index.db c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\Standard1-hc-1-Index.db,command error Code: 1, command output: Cannot create a file when that file already exists.
> [junit]
> [junit] at org.apache.cassandra.utils.CLibrary.exec(CLibrary.java:213)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:188)
> [junit] at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:151)
> [junit] at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
> [junit] at org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
> [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> [junit] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [junit] at java.lang.Thread.run(Thread.java:662)
> [junit] ERROR 17:10:17,111 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira