You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Anthony Molinaro (JIRA)" <ji...@apache.org> on 2010/01/21 07:38:54 UTC
[jira] Created: (CASSANDRA-730) When multiple DataFileDirectory
entries exist in config files for the same sstable can end up in different
directories.
When multiple DataFileDirectory entries exist in config files for the same sstable can end up in different directories.
-----------------------------------------------------------------------------------------------------------------------
Key: CASSANDRA-730
URL: https://issues.apache.org/jira/browse/CASSANDRA-730
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.5
Reporter: Anthony Molinaro
I have a cluster of 8 cassandra nodes, each node has 3 disks and I have the following in my storage-conf.xml
<DataFileDirectories>
<DataFileDirectory>/var/lib/cassandra/data1</DataFileDirectory>
<DataFileDirectory>/var/lib/cassandra/data2</DataFileDirectory>
<DataFileDirectory>/var/lib/cassandra/data3</DataFileDirectory>
</DataFileDirectories>
Upon adding a new node, setting AutoBootstrap to true and starting it, it seemed to start bootstrapping, but then
failed with the following exception.
java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
java.io.IOError: java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:154)
at org.apache.cassandra.io.SSTableWriter.renameAndOpen(SSTableWriter.java:161)
at org.apache.cassandra.io.Streaming$StreamCompletionHandler.onStreamCompletion(Streaming.java:301)
at org.apache.cassandra.net.io.ContentStreamState.handleStreamCompletion(ContentStreamState.java:108)
at org.apache.cassandra.net.io.ContentStreamState.read(ContentStreamState.java:90)
at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
at org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
at org.apache.cassandra.utils.FBUtilities.renameWithConfirm(FBUtilities.java:306)
at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:150)
... 9 more
Upon inspecting the directories I noticed the following
find /var/lib/cassandra -follow
/var/lib/cassandra
/var/lib/cassandra/data2
/var/lib/cassandra/data2/system
/var/lib/cassandra/data2/retarget
/var/lib/cassandra/data2/retarget/user_to_acctids-tmp-1-Filter.db
/var/lib/cassandra/.cassandra-placeholder
/var/lib/cassandra/data3
/var/lib/cassandra/data3/system
/var/lib/cassandra/data3/retarget
/var/lib/cassandra/data3/retarget/user_to_acctids-tmp-1-Data.db
/var/lib/cassandra/logs
/var/lib/cassandra/logs/system.log
/var/lib/cassandra/data1
/var/lib/cassandra/data1/system
/var/lib/cassandra/data1/retarget
/var/lib/cassandra/data1/retarget/user_to_acctids-tmp-1-Index.db
/var/lib/cassandra/commitlog
/var/lib/cassandra/commitlog/CommitLog-1264050238700.log
showing the sstable files were in different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-730) When multiple DataFileDirectory
entries exist in config files for the same sstable can end up in different
directories.
Posted by "Michael Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803239#action_12803239 ]
Michael Lee commented on CASSANDRA-730:
---------------------------------------
it's looks like same bug as CASSANDRA-716
> When multiple DataFileDirectory entries exist in config files for the same sstable can end up in different directories.
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-730
> URL: https://issues.apache.org/jira/browse/CASSANDRA-730
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Reporter: Anthony Molinaro
>
> I have a cluster of 8 cassandra nodes, each node has 3 disks and I have the following in my storage-conf.xml
> <DataFileDirectories>
> <DataFileDirectory>/var/lib/cassandra/data1</DataFileDirectory>
> <DataFileDirectory>/var/lib/cassandra/data2</DataFileDirectory>
> <DataFileDirectory>/var/lib/cassandra/data3</DataFileDirectory>
> </DataFileDirectories>
> Upon adding a new node, setting AutoBootstrap to true and starting it, it seemed to start bootstrapping, but then
> failed with the following exception.
> java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> java.io.IOError: java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:154)
> at org.apache.cassandra.io.SSTableWriter.renameAndOpen(SSTableWriter.java:161)
> at org.apache.cassandra.io.Streaming$StreamCompletionHandler.onStreamCompletion(Streaming.java:301)
> at org.apache.cassandra.net.io.ContentStreamState.handleStreamCompletion(ContentStreamState.java:108)
> at org.apache.cassandra.net.io.ContentStreamState.read(ContentStreamState.java:90)
> at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
> at org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> at org.apache.cassandra.utils.FBUtilities.renameWithConfirm(FBUtilities.java:306)
> at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:150)
> ... 9 more
> Upon inspecting the directories I noticed the following
> find /var/lib/cassandra -follow
> /var/lib/cassandra
> /var/lib/cassandra/data2
> /var/lib/cassandra/data2/system
> /var/lib/cassandra/data2/retarget
> /var/lib/cassandra/data2/retarget/user_to_acctids-tmp-1-Filter.db
> /var/lib/cassandra/.cassandra-placeholder
> /var/lib/cassandra/data3
> /var/lib/cassandra/data3/system
> /var/lib/cassandra/data3/retarget
> /var/lib/cassandra/data3/retarget/user_to_acctids-tmp-1-Data.db
> /var/lib/cassandra/logs
> /var/lib/cassandra/logs/system.log
> /var/lib/cassandra/data1
> /var/lib/cassandra/data1/system
> /var/lib/cassandra/data1/retarget
> /var/lib/cassandra/data1/retarget/user_to_acctids-tmp-1-Index.db
> /var/lib/cassandra/commitlog
> /var/lib/cassandra/commitlog/CommitLog-1264050238700.log
> showing the sstable files were in different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-730) When multiple DataFileDirectory
entries exist in config files for the same sstable can end up in different
directories.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-730.
--------------------------------------
Resolution: Duplicate
Ah, Michael is right. Closing this one so we track it in one place. Sorry Anthony. :)
> When multiple DataFileDirectory entries exist in config files for the same sstable can end up in different directories.
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-730
> URL: https://issues.apache.org/jira/browse/CASSANDRA-730
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Reporter: Anthony Molinaro
>
> I have a cluster of 8 cassandra nodes, each node has 3 disks and I have the following in my storage-conf.xml
> <DataFileDirectories>
> <DataFileDirectory>/var/lib/cassandra/data1</DataFileDirectory>
> <DataFileDirectory>/var/lib/cassandra/data2</DataFileDirectory>
> <DataFileDirectory>/var/lib/cassandra/data3</DataFileDirectory>
> </DataFileDirectories>
> Upon adding a new node, setting AutoBootstrap to true and starting it, it seemed to start bootstrapping, but then
> failed with the following exception.
> java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> java.io.IOError: java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:154)
> at org.apache.cassandra.io.SSTableWriter.renameAndOpen(SSTableWriter.java:161)
> at org.apache.cassandra.io.Streaming$StreamCompletionHandler.onStreamCompletion(Streaming.java:301)
> at org.apache.cassandra.net.io.ContentStreamState.handleStreamCompletion(ContentStreamState.java:108)
> at org.apache.cassandra.net.io.ContentStreamState.read(ContentStreamState.java:90)
> at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
> at org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> at org.apache.cassandra.utils.FBUtilities.renameWithConfirm(FBUtilities.java:306)
> at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:150)
> ... 9 more
> Upon inspecting the directories I noticed the following
> find /var/lib/cassandra -follow
> /var/lib/cassandra
> /var/lib/cassandra/data2
> /var/lib/cassandra/data2/system
> /var/lib/cassandra/data2/retarget
> /var/lib/cassandra/data2/retarget/user_to_acctids-tmp-1-Filter.db
> /var/lib/cassandra/.cassandra-placeholder
> /var/lib/cassandra/data3
> /var/lib/cassandra/data3/system
> /var/lib/cassandra/data3/retarget
> /var/lib/cassandra/data3/retarget/user_to_acctids-tmp-1-Data.db
> /var/lib/cassandra/logs
> /var/lib/cassandra/logs/system.log
> /var/lib/cassandra/data1
> /var/lib/cassandra/data1/system
> /var/lib/cassandra/data1/retarget
> /var/lib/cassandra/data1/retarget/user_to_acctids-tmp-1-Index.db
> /var/lib/cassandra/commitlog
> /var/lib/cassandra/commitlog/CommitLog-1264050238700.log
> showing the sstable files were in different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-730) When multiple DataFileDirectory
entries exist in config files for the same sstable can end up in different
directories.
Posted by "Anthony Molinaro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803417#action_12803417 ]
Anthony Molinaro commented on CASSANDRA-730:
--------------------------------------------
Not a problem, just glad it's on the radar, and hopefully it'll make it into 0.5.1.
By the way, a workaround when you get into this situation is to switch to a single directory in the config for the bootstrapping node and restart it.
Just stopping the bootstrapping node, actually puts the system into a broken state, where you constantly get these exceptions
ERROR [pool-1-thread-64] 2010-01-21 07:37:42,610 Cassandra.java (line 1064) Internal error processing insert
java.lang.AssertionError
at org.apache.cassandra.locator.TokenMetadata.getToken(TokenMetadata.java:212)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedMapForEndpoints(AbstractReplicationStrategy.java:129)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:76)
at org.apache.cassandra.service.StorageService.getHintedEndpointMap(StorageService.java:1186)
at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:169)
at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:466)
at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:417)
at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:1056)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
So I had to restart the bootstrapping node. This seems like it could be an issue if for whatever reason you start bootstrapping a node, then have to undo that action. Maybe there's the need for some sort of 'recover' command which helps you get out of a failed bootstrap?
> When multiple DataFileDirectory entries exist in config files for the same sstable can end up in different directories.
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-730
> URL: https://issues.apache.org/jira/browse/CASSANDRA-730
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Reporter: Anthony Molinaro
>
> I have a cluster of 8 cassandra nodes, each node has 3 disks and I have the following in my storage-conf.xml
> <DataFileDirectories>
> <DataFileDirectory>/var/lib/cassandra/data1</DataFileDirectory>
> <DataFileDirectory>/var/lib/cassandra/data2</DataFileDirectory>
> <DataFileDirectory>/var/lib/cassandra/data3</DataFileDirectory>
> </DataFileDirectories>
> Upon adding a new node, setting AutoBootstrap to true and starting it, it seemed to start bootstrapping, but then
> failed with the following exception.
> java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> java.io.IOError: java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:154)
> at org.apache.cassandra.io.SSTableWriter.renameAndOpen(SSTableWriter.java:161)
> at org.apache.cassandra.io.Streaming$StreamCompletionHandler.onStreamCompletion(Streaming.java:301)
> at org.apache.cassandra.net.io.ContentStreamState.handleStreamCompletion(ContentStreamState.java:108)
> at org.apache.cassandra.net.io.ContentStreamState.read(ContentStreamState.java:90)
> at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
> at org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> at org.apache.cassandra.utils.FBUtilities.renameWithConfirm(FBUtilities.java:306)
> at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:150)
> ... 9 more
> Upon inspecting the directories I noticed the following
> find /var/lib/cassandra -follow
> /var/lib/cassandra
> /var/lib/cassandra/data2
> /var/lib/cassandra/data2/system
> /var/lib/cassandra/data2/retarget
> /var/lib/cassandra/data2/retarget/user_to_acctids-tmp-1-Filter.db
> /var/lib/cassandra/.cassandra-placeholder
> /var/lib/cassandra/data3
> /var/lib/cassandra/data3/system
> /var/lib/cassandra/data3/retarget
> /var/lib/cassandra/data3/retarget/user_to_acctids-tmp-1-Data.db
> /var/lib/cassandra/logs
> /var/lib/cassandra/logs/system.log
> /var/lib/cassandra/data1
> /var/lib/cassandra/data1/system
> /var/lib/cassandra/data1/retarget
> /var/lib/cassandra/data1/retarget/user_to_acctids-tmp-1-Index.db
> /var/lib/cassandra/commitlog
> /var/lib/cassandra/commitlog/CommitLog-1264050238700.log
> showing the sstable files were in different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-730) When multiple DataFileDirectory
entries exist in config files for the same sstable can end up in different
directories.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803422#action_12803422 ]
Jonathan Ellis commented on CASSANDRA-730:
------------------------------------------
Yes, CASSANDRA-722 has a patch to fix that assertionerror.
> When multiple DataFileDirectory entries exist in config files for the same sstable can end up in different directories.
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-730
> URL: https://issues.apache.org/jira/browse/CASSANDRA-730
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Reporter: Anthony Molinaro
>
> I have a cluster of 8 cassandra nodes, each node has 3 disks and I have the following in my storage-conf.xml
> <DataFileDirectories>
> <DataFileDirectory>/var/lib/cassandra/data1</DataFileDirectory>
> <DataFileDirectory>/var/lib/cassandra/data2</DataFileDirectory>
> <DataFileDirectory>/var/lib/cassandra/data3</DataFileDirectory>
> </DataFileDirectories>
> Upon adding a new node, setting AutoBootstrap to true and starting it, it seemed to start bootstrapping, but then
> failed with the following exception.
> java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> java.io.IOError: java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:154)
> at org.apache.cassandra.io.SSTableWriter.renameAndOpen(SSTableWriter.java:161)
> at org.apache.cassandra.io.Streaming$StreamCompletionHandler.onStreamCompletion(Streaming.java:301)
> at org.apache.cassandra.net.io.ContentStreamState.handleStreamCompletion(ContentStreamState.java:108)
> at org.apache.cassandra.net.io.ContentStreamState.read(ContentStreamState.java:90)
> at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
> at org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: rename failed of /var/lib/cassandra/data3/retarget/user_to_acctids-1-Index.db
> at org.apache.cassandra.utils.FBUtilities.renameWithConfirm(FBUtilities.java:306)
> at org.apache.cassandra.io.SSTableWriter.rename(SSTableWriter.java:150)
> ... 9 more
> Upon inspecting the directories I noticed the following
> find /var/lib/cassandra -follow
> /var/lib/cassandra
> /var/lib/cassandra/data2
> /var/lib/cassandra/data2/system
> /var/lib/cassandra/data2/retarget
> /var/lib/cassandra/data2/retarget/user_to_acctids-tmp-1-Filter.db
> /var/lib/cassandra/.cassandra-placeholder
> /var/lib/cassandra/data3
> /var/lib/cassandra/data3/system
> /var/lib/cassandra/data3/retarget
> /var/lib/cassandra/data3/retarget/user_to_acctids-tmp-1-Data.db
> /var/lib/cassandra/logs
> /var/lib/cassandra/logs/system.log
> /var/lib/cassandra/data1
> /var/lib/cassandra/data1/system
> /var/lib/cassandra/data1/retarget
> /var/lib/cassandra/data1/retarget/user_to_acctids-tmp-1-Index.db
> /var/lib/cassandra/commitlog
> /var/lib/cassandra/commitlog/CommitLog-1264050238700.log
> showing the sstable files were in different directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.