You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "giovanni delussu (JIRA)" <ji...@apache.org> on 2012/11/22 07:32:58 UTC
[jira] [Created] (HADOOP-9084) TotalOrderPartitioner fails on
hadoop running on top of gpfs (or any parallel or distributed filesystem)
giovanni delussu created HADOOP-9084:
----------------------------------------
Summary: TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
Key: HADOOP-9084
URL: https://issues.apache.org/jira/browse/HADOOP-9084
Project: Hadoop Common
Issue Type: Bug
Components: filecache, fs, native
Affects Versions: 1.0.4, 1.0.3
Reporter: giovanni delussu
Priority: Critical
Fix For: 1.0.4, 1.0.3
When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9084) TotalOrderPartitioner fails on
hadoop running on top of gpfs (or any parallel or distributed filesystem)
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505360#comment-13505360 ]
Hudson commented on HADOOP-9084:
--------------------------------
Integrated in Hadoop-Yarn-trunk #50 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/50/])
HADOOP-9064. Ameding CHANGES.txt with correct JIRA ID, wrongfully used HADOOP-9084 before (tucu) (Revision 1414347)
HADOOP-9084. Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction. (kkambatl via tucu) (Revision 1414326)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414347
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414326
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegationTokenRenewer.java
> TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9084
> URL: https://issues.apache.org/jira/browse/HADOOP-9084
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache, fs, native
> Affects Versions: 1.0.3, 1.0.4
> Reporter: giovanni delussu
> Assignee: giovanni delussu
> Priority: Critical
> Fix For: 1.0.3, 1.0.4
>
> Attachments: PATCH-HADOOP-9084-origin-branch-1.0.patch
>
>
> When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9084) TotalOrderPartitioner fails on
hadoop running on top of gpfs (or any parallel or distributed filesystem)
Posted by "giovanni delussu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
giovanni delussu updated HADOOP-9084:
-------------------------------------
Attachment: PATCH-HADOOP-9084-origin-branch-1.0.patch
> TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9084
> URL: https://issues.apache.org/jira/browse/HADOOP-9084
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache, fs, native
> Affects Versions: 1.0.3, 1.0.4
> Reporter: giovanni delussu
> Assignee: giovanni delussu
> Priority: Critical
> Fix For: 1.0.3, 1.0.4
>
> Attachments: PATCH-HADOOP-9084-origin-branch-1.0.patch
>
>
> When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9084) TotalOrderPartitioner fails on
hadoop running on top of gpfs (or any parallel or distributed filesystem)
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504882#comment-13504882 ]
Hudson commented on HADOOP-9084:
--------------------------------
Integrated in Hadoop-trunk-Commit #3063 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3063/])
HADOOP-9064. Ameding CHANGES.txt with correct JIRA ID, wrongfully used HADOOP-9084 before (tucu) (Revision 1414347)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414347
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
> TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9084
> URL: https://issues.apache.org/jira/browse/HADOOP-9084
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache, fs, native
> Affects Versions: 1.0.3, 1.0.4
> Reporter: giovanni delussu
> Assignee: giovanni delussu
> Priority: Critical
> Fix For: 1.0.3, 1.0.4
>
> Attachments: PATCH-HADOOP-9084-origin-branch-1.0.patch
>
>
> When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9084) TotalOrderPartitioner fails on
hadoop running on top of gpfs (or any parallel or distributed filesystem)
Posted by "giovanni delussu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503667#comment-13503667 ]
giovanni delussu commented on HADOOP-9084:
------------------------------------------
the patch is intended for the branch1.0 (version 1.0.3-1.0.4) not for the trunk
> TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9084
> URL: https://issues.apache.org/jira/browse/HADOOP-9084
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache, fs, native
> Affects Versions: 1.0.3, 1.0.4
> Reporter: giovanni delussu
> Assignee: giovanni delussu
> Priority: Critical
> Fix For: 1.0.3, 1.0.4
>
> Attachments: PATCH-HADOOP-9084-origin-branch-1.0.patch
>
>
> When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9084) TotalOrderPartitioner fails on
hadoop running on top of gpfs (or any parallel or distributed filesystem)
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504868#comment-13504868 ]
Hudson commented on HADOOP-9084:
--------------------------------
Integrated in Hadoop-trunk-Commit #3062 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3062/])
HADOOP-9084. Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction. (kkambatl via tucu) (Revision 1414326)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414326
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegationTokenRenewer.java
> TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9084
> URL: https://issues.apache.org/jira/browse/HADOOP-9084
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache, fs, native
> Affects Versions: 1.0.3, 1.0.4
> Reporter: giovanni delussu
> Assignee: giovanni delussu
> Priority: Critical
> Fix For: 1.0.3, 1.0.4
>
> Attachments: PATCH-HADOOP-9084-origin-branch-1.0.patch
>
>
> When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9084) TotalOrderPartitioner fails on
hadoop running on top of gpfs (or any parallel or distributed filesystem)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502623#comment-13502623 ]
Hadoop QA commented on HADOOP-9084:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12554643/PATCH-HADOOP-9084-origin-branch-1.0.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1800//console
This message is automatically generated.
> TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9084
> URL: https://issues.apache.org/jira/browse/HADOOP-9084
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache, fs, native
> Affects Versions: 1.0.3, 1.0.4
> Reporter: giovanni delussu
> Assignee: giovanni delussu
> Priority: Critical
> Fix For: 1.0.3, 1.0.4
>
> Attachments: PATCH-HADOOP-9084-origin-branch-1.0.patch
>
>
> When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9084) TotalOrderPartitioner fails on
hadoop running on top of gpfs (or any parallel or distributed filesystem)
Posted by "giovanni delussu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
giovanni delussu updated HADOOP-9084:
-------------------------------------
Assignee: giovanni delussu
> TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9084
> URL: https://issues.apache.org/jira/browse/HADOOP-9084
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache, fs, native
> Affects Versions: 1.0.3, 1.0.4
> Reporter: giovanni delussu
> Assignee: giovanni delussu
> Priority: Critical
> Fix For: 1.0.3, 1.0.4
>
>
> When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9084) TotalOrderPartitioner fails on
hadoop running on top of gpfs (or any parallel or distributed filesystem)
Posted by "giovanni delussu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
giovanni delussu updated HADOOP-9084:
-------------------------------------
Status: Patch Available (was: Open)
*This patch works by keeping the fs and the jobconf working directories different. In hdfs we don't need it because the "local" fs is different from the default fs.
*potential problems of this patch: if a user creates a file ( with relative path) in the mapper/reducer phase this file will be created in the attempt directory and then deleted on cleanup. Without the patch the file would be created in the current working directory of the filesystem, i.e. the directory from which the user launched hadoop.
> TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9084
> URL: https://issues.apache.org/jira/browse/HADOOP-9084
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache, fs, native
> Affects Versions: 1.0.4, 1.0.3
> Reporter: giovanni delussu
> Assignee: giovanni delussu
> Priority: Critical
> Fix For: 1.0.4, 1.0.3
>
>
> When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira