You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2015/08/05 22:33:05 UTC
[jira] [Commented] (HADOOP-12057) swiftfs rename on partitioned
file attempts to consolidate partitions
[ https://issues.apache.org/jira/browse/HADOOP-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658851#comment-14658851 ]
Steve Loughran commented on HADOOP-12057:
-----------------------------------------
All the rename tests are failing for me now
{code}
------------------------------------------------------
Running org.apache.hadoop.fs.swift.TestSwiftFileSystemRename
Tests run: 14, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 187.855 sec <<< FAILURE! - in org.apache.hadoop.fs.swift.TestSwiftFileSystemRename
testRenameFileAndVerifyContents(org.apache.hadoop.fs.swift.TestSwiftFileSystemRename) Time elapsed: 29.876 sec <<< ERROR!
java.io.FileNotFoundException: Operation COPY on https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_8b95cbaa-d1b8-43a3-97f8-dbc8d4b3174a/jenkins/test/home/user/documents/file.txt/file.txt
at org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1449)
at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1402)
at org.apache.hadoop.fs.swift.http.SwiftRestClient.copyObject(SwiftRestClient.java:923)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.copyObject(SwiftNativeFileSystemStore.java:815)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.movePartitionedFile(SwiftNativeFileSystemStore.java:746)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.rename(SwiftNativeFileSystemStore.java:628)
at org.apache.hadoop.fs.swift.SwiftFileSystemBaseTest.renameToSuccess(SwiftFileSystemBaseTest.java:283)
at org.apache.hadoop.fs.swift.SwiftFileSystemBaseTest.rename(SwiftFileSystemBaseTest.java:227)
at org.apache.hadoop.fs.swift.TestSwiftFileSystemRename.testRenameFileAndVerifyContents(TestSwiftFileSystemRename.java:197)
testRenameFile(org.apache.hadoop.fs.swift.TestSwiftFileSystemRename) Time elapsed: 20.657 sec <<< ERROR!
java.io.FileNotFoundException: Operation COPY on https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_8b95cbaa-d1b8-43a3-97f8-dbc8d4b3174a/jenkins/test/alice/file/file
at org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1449)
at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1402)
at org.apache.hadoop.fs.swift.http.SwiftRestClient.copyObject(SwiftRestClient.java:923)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.copyObject(SwiftNativeFileSystemStore.java:815)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.movePartitionedFile(SwiftNativeFileSystemStore.java:746)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.rename(SwiftNativeFileSystemStore.java:628)
at org.apache.hadoop.fs.swift.SwiftFileSystemBaseTest.renameToSuccess(SwiftFileSystemBaseTest.java:283)
at org.apache.hadoop.fs.swift.SwiftFileSystemBaseTest.rename(SwiftFileSystemBaseTest.java:227)
at org.apache.hadoop.fs.swift.TestSwiftFileSystemRename.testRenameFile(TestSwiftFileSystemRename.java:76)
testRenamedConsistence(org.apache.hadoop.fs.swift.TestSwiftFileSystemRename) Time elapsed: 24.578 sec <<< ERROR!
java.io.FileNotFoundException: Operation COPY on https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_8b95cbaa-d1b8-43a3-97f8-dbc8d4b3174a/jenkins/test/home/user/documents/file.txt/file.txt
at org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1449)
at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1402)
at org.apache.hadoop.fs.swift.http.SwiftRestClient.copyObject(SwiftRestClient.java:923)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.copyObject(SwiftNativeFileSystemStore.java:815)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.movePartitionedFile(SwiftNativeFileSystemStore.java:746)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.rename(SwiftNativeFileSystemStore.java:628)
at org.apache.hadoop.fs.swift.SwiftFileSystemBaseTest.renameToSuccess(SwiftFileSystemBaseTest.java:283)
at org.apache.hadoop.fs.swift.SwiftFileSystemBaseTest.rename(SwiftFileSystemBaseTest.java:227)
at org.apache.hadoop.fs.swift.TestSwiftFileSystemRename.testRenamedConsistence(TestSwiftFileSystemRename.java:259)
testRenameFileIntoExistingDirectory(org.apache.hadoop.fs.swift.TestSwiftFileSystemRename) Time elapsed: 26.019 sec <<< ERROR!
java.io.FileNotFoundException: Operation COPY on https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_8b95cbaa-d1b8-43a3-97f8-dbc8d4b3174a/jenkins/test/olddir/file/file
at org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1449)
at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1402)
at org.apache.hadoop.fs.swift.http.SwiftRestClient.copyObject(SwiftRestClient.java:923)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.copyObject(SwiftNativeFileSystemStore.java:815)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.movePartitionedFile(SwiftNativeFileSystemStore.java:746)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.rename(SwiftNativeFileSystemStore.java:628)
at org.apache.hadoop.fs.swift.SwiftFileSystemBaseTest.renameToSuccess(SwiftFileSystemBaseTest.java:283)
at org.apache.hadoop.fs.swift.SwiftFileSystemBaseTest.rename(SwiftFileSystemBaseTest.java:227)
at org.apache.hadoop.fs.swift.TestSwiftFileSystemRename.testRenameFileIntoExistingDirectory(TestSwiftFileSystemRename.java:51)
{code}
> swiftfs rename on partitioned file attempts to consolidate partitions
> ---------------------------------------------------------------------
>
> Key: HADOOP-12057
> URL: https://issues.apache.org/jira/browse/HADOOP-12057
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/swift
> Reporter: David Dobbins
> Assignee: David Dobbins
> Attachments: HADOOP-12057.patch, HADOOP-12057.patch, HADOOP-12057.patch, HADOOP-12057.patch, HADOOP-12057.patch
>
>
> In the swift filesystem for openstack, a rename operation on a partitioned file uses the swift COPY operation, which attempts to consolidate all of the partitions into a single object. This causes the rename to fail when the total size of all the partitions exceeds the maximum object size for swift. Since partitioned files are primarily created to allow a file to exceed the maximum object size, this bug makes writing to swift extremely unreliable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)