You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Ahad Rana (JIRA)" <ji...@apache.org> on 2007/08/25 10:09:30 UTC
[jira] Created: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
keyToPath in Jets3tFileSystemStore needs to return absolute path
----------------------------------------------------------------
Key: HADOOP-1783
URL: https://issues.apache.org/jira/browse/HADOOP-1783
Project: Hadoop
Issue Type: Bug
Components: fs/s3
Affects Versions: 0.14.0, 0.13.1, 0.13.0, 0.12.3, 0.12.2, 0.12.1, 0.12.0, 0.11.2, 0.11.1, 0.11.0, 0.10.1, 0.10.0, 0.9.2, 0.9.1, 0.9.0, 0.8.0, 0.7.2, 0.7.1, 0.7.0, 0.6.2, 0.6.1, 0.6.0, 0.5.0, 0.4.0, 0.3.2, 0.3.1, 0.3.0, 0.2.1, 0.2.0, 0.1.1, 0.1.0
Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
Reporter: Ahad Rana
The keyToPath method probably needs to:
1. take the bucket identifier as a parameter.
2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-1783:
------------------------------
Fix Version/s: 0.15.0
0.14.2
Affects Version/s: (was: 0.11.2)
(was: 0.11.1)
(was: 0.11.0)
(was: 0.10.1)
(was: 0.10.0)
(was: 0.9.2)
(was: 0.9.1)
(was: 0.9.0)
(was: 0.8.0)
(was: 0.7.2)
(was: 0.7.1)
(was: 0.7.0)
(was: 0.6.2)
(was: 0.6.1)
(was: 0.6.0)
(was: 0.5.0)
(was: 0.4.0)
(was: 0.3.2)
(was: 0.3.1)
(was: 0.3.0)
(was: 0.2.1)
(was: 0.2.0)
(was: 0.1.1)
(was: 0.1.0)
0.14.1
Status: Patch Available (was: Open)
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.14.1, 0.14.0, 0.13.1, 0.13.0, 0.12.3, 0.12.2, 0.12.1, 0.12.0
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
> Assignee: Tom White
> Fix For: 0.14.2, 0.15.0
>
> Attachments: HADOOP-1783.patch
>
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12524876 ]
Doug Cutting commented on HADOOP-1783:
--------------------------------------
> I believe it would be a good thing to make it a rule that all FileSystem implementations return fully qualified (scheme and host specified) paths in all cases.
This sounds reasonable to me. Could you submit a new issue for it? I think the fix would be to (1) note this in FileSystem's javadoc; (1) fix all implementations; (3) add some unit tests.
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.5.0, 0.6.0, 0.6.1, 0.6.2, 0.7.0, 0.7.1, 0.7.2, 0.8.0, 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.10.1, 0.11.0, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
> Attachments: HADOOP-1783.patch
>
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-1783:
------------------------------
Attachment: HADOOP-1783.patch
Hi Ahad,
bq. But, ultimately, if you look at the DistributedFileSystem implementation of listPaths, it clearly creates fully qualified paths using the DfsPath(DFSFileInfo,FileSystem) constructor.
Yes - I missed this.
bq. In the case of the s3 implementation, the listPaths, as I mentioned, returns sub-paths without the scheme or the bucket name (authorization).
I've created a patch that fixes this (in the same way as DistributedFileSystem). Does this solve your problem?
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.5.0, 0.6.0, 0.6.1, 0.6.2, 0.7.0, 0.7.1, 0.7.2, 0.8.0, 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.10.1, 0.11.0, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
> Attachments: HADOOP-1783.patch
>
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12524174 ]
Tom White commented on HADOOP-1783:
-----------------------------------
All paths in Jets3tFileSystemStore are absolute - they begin with '/' since this is enforced in S3FileSystem by the makeAbsolute method. As you point out, they do not include scheme and authority parts. However, I believe this is in line with HDFS, where DistributedFileSystem makes sure paths are absolute but doesn't include scheme and authority parts.
What is the problem that you have encountered - could you give more details (e.g. set up, stacktrace) please?
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.5.0, 0.6.0, 0.6.1, 0.6.2, 0.7.0, 0.7.1, 0.7.2, 0.8.0, 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.10.1, 0.11.0, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Ahad Rana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12524239 ]
Ahad Rana commented on HADOOP-1783:
-----------------------------------
On a broader note, I believe it would be a good thing to make it a rule that all FileSystem implementations return fully qualified (scheme and host specified) paths in all cases. Since the system uses Path objects everywhere, and in most cases does a late, on demand, resolution from Path to FileSystem object and then Input / OutputStream, an improperly qualified path combined with the use of more than one FileSystem for an Input / Output source, is bound to cause unpredictable behavior elsewhere in the code as well.
Just a suggestion.
Thanks,
Ahad.
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.5.0, 0.6.0, 0.6.1, 0.6.2, 0.7.0, 0.7.1, 0.7.2, 0.8.0, 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.10.1, 0.11.0, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-1783:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've just committed this.
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0, 0.14.1
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
> Assignee: Tom White
> Fix For: 0.14.2, 0.15.0
>
> Attachments: HADOOP-1783-branch-0.14.patch, HADOOP-1783.patch
>
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528126 ]
Tom White commented on HADOOP-1783:
-----------------------------------
I've just added HADOOP-1909 for the fix for other implementations.
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0, 0.14.1
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
> Assignee: Tom White
> Fix For: 0.14.2, 0.15.0
>
> Attachments: HADOOP-1783.patch
>
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528482 ]
Hadoop QA commented on HADOOP-1783:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12364948/HADOOP-1783.patch
against trunk revision r576315.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/781/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/781/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/781/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/781/console
This message is automatically generated.
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0, 0.14.1
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
> Assignee: Tom White
> Fix For: 0.14.2, 0.15.0
>
> Attachments: HADOOP-1783.patch
>
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529841 ]
Hudson commented on HADOOP-1783:
--------------------------------
Integrated in Hadoop-Nightly #248 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/248/])
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0, 0.14.1
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
> Assignee: Tom White
> Fix For: 0.14.2, 0.15.0
>
> Attachments: HADOOP-1783-branch-0.14.patch, HADOOP-1783.patch
>
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Ahad Rana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525530 ]
Ahad Rana commented on HADOOP-1783:
-----------------------------------
Hi Tom,
The patch looks good to me. Are you going to add the JIRA issue related to updating the FileSystem's javadocs ?
Thx.
Ahad.
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.5.0, 0.6.0, 0.6.1, 0.6.2, 0.7.0, 0.7.1, 0.7.2, 0.8.0, 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.10.1, 0.11.0, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
> Attachments: HADOOP-1783.patch
>
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-1783:
------------------------------
Attachment: HADOOP-1783-branch-0.14.patch
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0, 0.14.1
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
> Assignee: Tom White
> Fix For: 0.14.2, 0.15.0
>
> Attachments: HADOOP-1783-branch-0.14.patch, HADOOP-1783.patch
>
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White reassigned HADOOP-1783:
---------------------------------
Assignee: Tom White
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0, 0.14.1
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
> Assignee: Tom White
> Fix For: 0.14.2, 0.15.0
>
> Attachments: HADOOP-1783.patch
>
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1783) keyToPath in Jets3tFileSystemStore
needs to return absolute path
Posted by "Ahad Rana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12524238 ]
Ahad Rana commented on HADOOP-1783:
-----------------------------------
Hi Tom,
I will try to produce some stack traces for you. But, ultimately, if you look at the DistributedFileSystem implementation of listPaths, it clearly creates fully qualified paths using the DfsPath(DFSFileInfo,FileSystem) constructor. In the case of the s3 implementation, the listPaths, as I mentioned, returns sub-paths without the scheme or the bucket name (authorization). If the default file system is not s3, then the hadoop library returns improper results by trying to resolve the returned sub-path against the default FileSystem ( since the scheme is missing from the path object).
I am working on enabling map-reduce functionality for scenarios where either both, or at least one file specification (map input, and reduce output) in a map reduce spec points to the s3 file system. The above mentioned bug breaks the code in a couple of different places. When I implement keytoPath in Jet3FileSystemStore as follows, everything works.
private Path keyToPath(String key) {
return new Path("s3://"+bucket.getName()+key);
}
Suffice it to say, there are other (performance related) issues that I am also looking at in order to enable satisfactory use of s3 as a potential input/output for a mapreduce job. But, by far, this bug is the most critically broken issue.
Sorry about the lack of stack traces. I just need to recreate a proper test environment to get you these, and hopefully I will be able to submit something to you next week.
Thanks,
Ahad.
> keyToPath in Jets3tFileSystemStore needs to return absolute path
> ----------------------------------------------------------------
>
> Key: HADOOP-1783
> URL: https://issues.apache.org/jira/browse/HADOOP-1783
> Project: Hadoop
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.5.0, 0.6.0, 0.6.1, 0.6.2, 0.7.0, 0.7.1, 0.7.2, 0.8.0, 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.10.1, 0.11.0, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.13.1, 0.14.0
> Environment: hadoop 0.14.0 running under ec2 with s3 filesystem
> Reporter: Ahad Rana
>
> The keyToPath method probably needs to:
> 1. take the bucket identifier as a parameter.
> 2. set the returned Path object's protocol plus authority (bucket). Currently, APIs such as <i>listSubPaths</i> return relative paths (for a directory listing). This in turn breaks map reduce operations if the default file system is set to be something other than S3 (via fs.default.name, for example).
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.