You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Harsh J (JIRA)" <ji...@apache.org> on 2011/09/20 09:06:09 UTC
[jira] [Created] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
--------------------------------------------------------------------
Key: HADOOP-7659
URL: https://issues.apache.org/jira/browse/HADOOP-7659
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 0.20.204.0
Reporter: Harsh J
Priority: Minor
Fix For: 0.24.0
When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-7659:
----------------------------
Attachment: HADOOP-7659.patch
Going the doc way.
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Harsh J (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J reassigned HADOOP-7659:
-------------------------------
Assignee: Harsh J
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Fix For: 0.24.0
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-7659:
----------------------------
Target Version/s: 3.0.0
Fix Version/s: (was: 0.24.0)
Given that this is a trivial change that merely adds a helpful javadoc note, if no one has any comments on this, am going to commit it in by monday.
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283504#comment-13283504 ]
Hudson commented on HADOOP-7659:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2307 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2307/])
HADOOP-7659. fs -getmerge isn't guaranteed to work well over non-HDFS filesystems (harsh) (Revision 1342600)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342600
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167198#comment-13167198 ]
Hadoop QA commented on HADOOP-7659:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506909/HADOOP-7659.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated 5 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/465//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/465//console
This message is automatically generated.
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283405#comment-13283405 ]
Hudson commented on HADOOP-7659:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1090 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1090/])
HADOOP-7659. fs -getmerge isn't guaranteed to work well over non-HDFS filesystems (harsh) (Revision 1342600)
Result = ABORTED
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342600
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283432#comment-13283432 ]
Hudson commented on HADOOP-7659:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2361 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2361/])
HADOOP-7659. fs -getmerge isn't guaranteed to work well over non-HDFS filesystems (harsh) (Revision 1342600)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342600
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-7659:
----------------------------
Issue Type: Improvement (was: Bug)
In hindsight, isn't a bug but rather an improvement (i.e. we can document what to expect)
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283348#comment-13283348 ]
Harsh J commented on HADOOP-7659:
---------------------------------
bq. -1 javadoc. The javadoc tool appears to have generated 5 warning messages.
Was probably something else in trunk at the time. See command log below for {{mvn javadoc:javadoc}}, which I made sure to do again now before committing:
{code}
➜ trunk svn diff
Index: hadoop-common-project/hadoop-common/CHANGES.txt
===================================================================
--- hadoop-common-project/hadoop-common/CHANGES.txt (revision 1342586)
+++ hadoop-common-project/hadoop-common/CHANGES.txt (working copy)
@@ -76,6 +76,9 @@
HADOOP-8415. Add getDouble() and setDouble() in
org.apache.hadoop.conf.Configuration (Jan van der Lugt via harsh)
+ HADOOP-7659. fs -getmerge isn't guaranteed to work well over non-HDFS
+ filesystems (harsh)
+
BUG FIXES
HADOOP-8177. MBeans shouldn't try to register when it fails to create MBeanName.
Index: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
===================================================================
--- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java(revision 1342586)
+++ hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java(working copy)
@@ -307,6 +307,12 @@
return FileUtil.fullyDelete(f);
}
+ /**
+ * {@inheritDoc}
+ *
+ * (<b>Note</b>: Returned list is not sorted in any given order,
+ * due to reliance on Java's {@link File#list()} API.)
+ */
public FileStatus[] listStatus(Path f) throws IOException {
File localf = pathToFile(f);
FileStatus[] results;
➜ trunk cd hadoop-common-project/hadoop-common
➜ hadoop-common mvn javadoc:javadoc
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop Common 3.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> maven-javadoc-plugin:2.8.1:javadoc (default-cli) @ hadoop-common >>>
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-common ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @ hadoop-common ---
[INFO] Source directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/generated-sources/java added.
[INFO]
[INFO] --- build-helper-maven-plugin:1.5:add-test-source (add-test-source) @ hadoop-common ---
[INFO] Test Source directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java added.
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run (compile-proto) @ hadoop-common ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run (save-version) @ hadoop-common ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-dependency-plugin:2.1:build-classpath (build-classpath) @ hadoop-common ---
[INFO] Skipped writing classpath file '/Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/classes/mrapp-generated-classpath'. No changes found.
[INFO]
[INFO] <<< maven-javadoc-plugin:2.8.1:javadoc (default-cli) @ hadoop-common <<<
[INFO]
[INFO] --- maven-javadoc-plugin:2.8.1:javadoc (default-cli) @ hadoop-common ---
[INFO]
ExcludePrivateAnnotationsStandardDoclet
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23.042s
[INFO] Finished at: Fri May 25 18:14:53 GMT+05:30 2012
[INFO] Final Memory: 11M/81M
[INFO] ------------------------------------------------------------------------
{code}
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283444#comment-13283444 ]
Hudson commented on HADOOP-7659:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #2288 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2288/])
HADOOP-7659. fs -getmerge isn't guaranteed to work well over non-HDFS filesystems (harsh) (Revision 1342600)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342600
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283962#comment-13283962 ]
Hudson commented on HADOOP-7659:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #1057 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1057/])
HADOOP-7659. fs -getmerge isn't guaranteed to work well over non-HDFS filesystems (harsh) (Revision 1342600)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342600
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-7659:
----------------------------
Status: Patch Available (was: Open)
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7659) fs -getmerge isn't guaranteed to
work well over non-HDFS filesystems
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HADOOP-7659:
----------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Target Version/s: (was: 3.0.0)
Release Note: Documented that the "fs -getmerge" shell command may not work properly over non HDFS-filesystem implementations due to platform-varying file list ordering.
Status: Resolved (was: Patch Available)
Committed revision 1342600 to trunk.
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.20.204.0
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000, part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter only makes it worthy for what we include - while other FSes out there still have to deal with this issue. Perhaps we need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira