You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Jagane Sundar (Created) (JIRA)" <ji...@apache.org> on 2012/03/04 08:40:15 UTC
[jira] [Created] (HADOOP-8136) Enhance hadoop to use a newer
version (0.8.1) of the jets3t library
Enhance hadoop to use a newer version (0.8.1) of the jets3t library
-------------------------------------------------------------------
Key: HADOOP-8136
URL: https://issues.apache.org/jira/browse/HADOOP-8136
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Affects Versions: 1.0.0, 0.22.0, 0.23.3
Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
Reporter: Jagane Sundar
Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.
Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.
It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8136) Enhance hadoop to use a newer
version (0.8.1) of the jets3t library
Posted by "Jagane Sundar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jagane Sundar updated HADOOP-8136:
----------------------------------
Attachment: HADOOP-8136-0-for_branch_1_0.patch
Apply the patch HADOOP-5836-for_branch_1_0.patch before applying this patch.
> Enhance hadoop to use a newer version (0.8.1) of the jets3t library
> -------------------------------------------------------------------
>
> Key: HADOOP-8136
> URL: https://issues.apache.org/jira/browse/HADOOP-8136
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 0.22.0, 1.0.0, 0.23.3
> Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
> Reporter: Jagane Sundar
> Attachments: HADOOP-8136-0-for_branch_1_0.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
> The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.
> Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.
> It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8136) Enhance hadoop to use a newer
version (0.8.1) of the jets3t library
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8136:
-------------------------------
Target Version/s: 1.1.0 (was: 1.0.3)
Not resolved in time for 1.0.3. Please consider continuing the contribution in 1.1.0. Thank you.
> Enhance hadoop to use a newer version (0.8.1) of the jets3t library
> -------------------------------------------------------------------
>
> Key: HADOOP-8136
> URL: https://issues.apache.org/jira/browse/HADOOP-8136
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 1.0.0, 0.22.0, 0.23.3
> Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
> Reporter: Jagane Sundar
> Attachments: HADOOP-8136-0-for_branch_1_0.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
> The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.
> Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.
> It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8136) Enhance hadoop to use a newer
version (0.8.1) of the jets3t library
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-8136:
-------------------------------
Target Version/s: 1.0.3 (was: 1.0.2)
> Enhance hadoop to use a newer version (0.8.1) of the jets3t library
> -------------------------------------------------------------------
>
> Key: HADOOP-8136
> URL: https://issues.apache.org/jira/browse/HADOOP-8136
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 0.22.0, 1.0.0, 0.23.3
> Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
> Reporter: Jagane Sundar
> Attachments: HADOOP-8136-0-for_branch_1_0.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
> The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.
> Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.
> It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8136) Enhance hadoop to use a newer
version (0.8.1) of the jets3t library
Posted by "Matt Foley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232215#comment-13232215 ]
Matt Foley commented on HADOOP-8136:
------------------------------------
Jagane, please attach patch(es) for trunk, and enable auto-test via "Patch Available", to assure no side effects of the version upgrade. Thanks.
> Enhance hadoop to use a newer version (0.8.1) of the jets3t library
> -------------------------------------------------------------------
>
> Key: HADOOP-8136
> URL: https://issues.apache.org/jira/browse/HADOOP-8136
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 0.22.0, 1.0.0, 0.23.3
> Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
> Reporter: Jagane Sundar
> Attachments: HADOOP-8136-0-for_branch_1_0.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
> The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.
> Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.
> It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8136) Enhance hadoop to use a newer
version (0.8.1) of the jets3t library
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223008#comment-13223008 ]
Hadoop QA commented on HADOOP-8136:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517196/HADOOP-8136-0-for_branch_1_0.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/679//console
This message is automatically generated.
> Enhance hadoop to use a newer version (0.8.1) of the jets3t library
> -------------------------------------------------------------------
>
> Key: HADOOP-8136
> URL: https://issues.apache.org/jira/browse/HADOOP-8136
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 0.22.0, 1.0.0, 0.23.3
> Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
> Reporter: Jagane Sundar
> Attachments: HADOOP-8136-0-for_branch_1_0.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
> The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.
> Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.
> It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8136) Enhance hadoop to use a newer
version (0.8.1) of the jets3t library
Posted by "Jagane Sundar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jagane Sundar updated HADOOP-8136:
----------------------------------
Target Version/s: 1.0.2
Status: Patch Available (was: Open)
Here's my first cut at a patch for this jira. Some notes:
1. Most of the changes are on account of the jets3t-0.8.1 library throwing a ServiceException, instead of a S3ServiceException.
2. There is also a piece of brittle code that was comparing an exception string with 'ResponseCode=404' to decide whether the S3 server threw a 404 error. I replaced that with a more rational test of response code.
3. I turned down the logging by setting
org.apache.hadoop.fs.s3native=ERROR
org.jets3t.service.impl.rest.httpclient.RestStorageService=ERROR
in log4j.properties
> Enhance hadoop to use a newer version (0.8.1) of the jets3t library
> -------------------------------------------------------------------
>
> Key: HADOOP-8136
> URL: https://issues.apache.org/jira/browse/HADOOP-8136
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 1.0.0, 0.22.0, 0.23.3
> Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
> Reporter: Jagane Sundar
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
> The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.
> Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.
> It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8136) Enhance hadoop to use a newer
version (0.8.1) of the jets3t library
Posted by "Jagane Sundar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232297#comment-13232297 ]
Jagane Sundar commented on HADOOP-8136:
---------------------------------------
I am working on the patch for trunk, Matt. Running into mvn difficulties. Mvn is picking up the old version of jets3t. Once I fix that, I should have the trunk patch ready. Hopefully later today.
Jagane
----- Reply message -----
From: "Matt Foley (Commented) (JIRA)" <ji...@apache.org>
To: "Jagane Sundar" <ja...@sundar.org>
Subject: [jira] [Commented] (HADOOP-8136) Enhance hadoop to use a newer version (0.8.1) of the jets3t library
Date: Sun, Mar 18, 2012 1:20 am
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232215#comment-13232215 ]
Matt Foley commented on HADOOP-8136:
------------------------------------
Jagane, please attach patch(es) for trunk, and enable auto-test via "Patch Available", to assure no side effects of the version upgrade. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
> Enhance hadoop to use a newer version (0.8.1) of the jets3t library
> -------------------------------------------------------------------
>
> Key: HADOOP-8136
> URL: https://issues.apache.org/jira/browse/HADOOP-8136
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 0.22.0, 1.0.0, 0.23.3
> Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
> Reporter: Jagane Sundar
> Attachments: HADOOP-8136-0-for_branch_1_0.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
> The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.
> Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.
> It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HADOOP-8136) Enhance hadoop to use a
newer version (0.8.1) of the jets3t library
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232297#comment-13232297 ]
Matt Foley edited comment on HADOOP-8136 at 5/7/12 1:19 AM:
------------------------------------------------------------
I am working on the patch for trunk, Matt. Running into mvn difficulties. Mvn is picking up the old version of jets3t. Once I fix that, I should have the trunk patch ready. Hopefully later today.
Jagane
was (Author: jagane):
I am working on the patch for trunk, Matt. Running into mvn difficulties. Mvn is picking up the old version of jets3t. Once I fix that, I should have the trunk patch ready. Hopefully later today.
Jagane
----- Reply message -----
From: "Matt Foley (Commented) (JIRA)" <ji...@apache.org>
To: "Jagane Sundar" <ja...@sundar.org>
Subject: [jira] [Commented] (HADOOP-8136) Enhance hadoop to use a newer version (0.8.1) of the jets3t library
Date: Sun, Mar 18, 2012 1:20 am
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232215#comment-13232215 ]
Matt Foley commented on HADOOP-8136:
------------------------------------
Jagane, please attach patch(es) for trunk, and enable auto-test via "Patch Available", to assure no side effects of the version upgrade. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
> Enhance hadoop to use a newer version (0.8.1) of the jets3t library
> -------------------------------------------------------------------
>
> Key: HADOOP-8136
> URL: https://issues.apache.org/jira/browse/HADOOP-8136
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 1.0.0, 0.22.0, 0.23.3
> Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
> Reporter: Jagane Sundar
> Attachments: HADOOP-8136-0-for_branch_1_0.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
> The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.
> Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.
> It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8136) Enhance hadoop to use a newer
version (0.8.1) of the jets3t library
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230716#comment-13230716 ]
Suresh Srinivas commented on HADOOP-8136:
-----------------------------------------
Jagane, can you attach a patch based on trunk, so Jenkins can apply the patch and test it?
> Enhance hadoop to use a newer version (0.8.1) of the jets3t library
> -------------------------------------------------------------------
>
> Key: HADOOP-8136
> URL: https://issues.apache.org/jira/browse/HADOOP-8136
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 0.22.0, 1.0.0, 0.23.3
> Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
> Reporter: Jagane Sundar
> Attachments: HADOOP-8136-0-for_branch_1_0.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Hadoop is built against, and includes, an older version of the Jets3t library - version 0.6.1.
> The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version includes many improvements such as support for "Requester-Pays" buckets.
> Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with an error stating that the method signature of some method in the Jets3t library does not match.
> It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published by the 0.8.1 version of the jets3t library.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira