You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Tom White (Created) (JIRA)" <ji...@apache.org> on 2012/04/13 01:45:17 UTC
[jira] [Created] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
MapReduce should not have a compile-time dependency on HDFS
-----------------------------------------------------------
Key: MAPREDUCE-4148
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tom White
Assignee: Tom White
MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Attachment: MAPREDUCE-4148.patch
Thanks for the comments. This latest patch fixes decodeIdentifier to check for null.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed MAPREDUCE-4148.
------------------------------------
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.2-alpha
>
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268460#comment-13268460 ]
Daryn Sharp commented on MAPREDUCE-4148:
----------------------------------------
I'd like to suggest caching the service loader lookups since there will be a fair amount of decoding.
I'd like to see the kind and service printed before the decoded identifier. The vast majority of the time, it's those two fields that are critical to debugging. The ident will likely cause those to wrap to the next line.
Otherwise, looks great!!
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4148:
-------------------------------------------
Status: Open (was: Patch Available)
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Status: Patch Available (was: Open)
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Attachment: MAPREDUCE-4148.patch
New patch addressing Daryn's two points.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272520#comment-13272520 ]
Robert Joseph Evans commented on MAPREDUCE-4148:
------------------------------------------------
I am +1 for the changes too, assuming that Jenkins Comes back OK.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273369#comment-13273369 ]
Hudson commented on MAPREDUCE-4148:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2247 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2247/])
MAPREDUCE-4148. MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199)
Result = ABORTED
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.0
>
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268131#comment-13268131 ]
Hadoop QA commented on MAPREDUCE-4148:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12525562/MAPREDUCE-4148.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2355//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2355//console
This message is automatically generated.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Component/s: mrv2
Fix Version/s: 2.0.0
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.0
>
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262139#comment-13262139 ]
Daryn Sharp commented on MAPREDUCE-4148:
----------------------------------------
Very nice! I've been wanting to do something similar to allow tokens to decode their identifiers for quite awhile now. Thoughts/suggestions:
{{Token#decodeIdentifier()}} would be _really_ useful and prevent callers from having to know about the factory. This would very nearly abstract away the details. It could either delegate to the factory as-is, or just query the factory for the class, or maybe even {{Token}} can host the kind/class registration method. There are a number of other places in the code, ex. {{AbstractDelegationTokenSecretManager#renewToken(Token)}} that could benefit from such a method.
I'm not too fond of {{AbstractDelegationTokenIdentifier#stringifyToken(Token)}}. It creates a circular relationship. A {{TokenIdentifier}} really shouldn't have to know about a {{Token}} wrapper. How removing the inversion of knowledge, and update {{Token#toString()}} to use {{token.decodeIdentifier()}}? If the value is null because the class isn't available, it can print the raw bytes like it does now.
{{TokenIdentifierFactory.createIdentifier}} is using {{ReflectionUtils.newInstance(Configuration)}} whose main purpose is to invoke {{setConf(conf)}} and/or MR's {{configure(conf)}} which isn't applicable in this case. How about directly using {{class.newInstance()}}?
I think there's pitfalls with using static class inits for the factory registration. The identifier class has to be loaded (sorry if I'm stating the obvious: not just imported, but something referenced from it) before a token can be decoded. In general this probably means a token has to be created before other tokens can be decoded. Perhaps the static blocks should become a static class method that's invoked by the secret manager, since we know the secret manager is instantiated before token manipulation. Although, that limits token ident decoding only to tokens owned by the daemon, which would leave a client out of luck. Maybe you could get fancy with a class loader.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269917#comment-13269917 ]
Hadoop QA commented on MAPREDUCE-4148:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12525882/MAPREDUCE-4148.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
org.apache.hadoop.ipc.TestSaslRPC
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2361//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2361//console
This message is automatically generated.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274010#comment-13274010 ]
Hudson commented on MAPREDUCE-4148:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1077 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/])
MAPREDUCE-4148. MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199)
Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.0
>
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268698#comment-13268698 ]
Daryn Sharp commented on MAPREDUCE-4148:
----------------------------------------
Just a question: is there an advantage to using google's maps? Given this jira's title, it seems ironic to add a dep. :)
I've been discouraged from using static class init blocks due to the wonky exceptions it may cause. I'd consider having something like a static "getClassForIdentifier" that populates a map on first call from {{decodeIdentifier}}.
I missed it before, but you can pass a null conf to {{ReflectionUtils.newInstance}}. It shorts out trying to set the conf when null, and it also avoids instantiating unnecessary objects. Configs also hang around in a weak hash map for awhile...
Mainly because redundancy drives me nuts, but also because some errant exception could cause {{addBinaryBuffer}} to be called twice, would you consider this?
{code}
private void identifierToString(StringBuilder buffer) {
T id = null;
try {
id = decodeIdentifier();
} finally {
if (id != null) {
buffer.append("(").append(id).append(")");
} else {
addBinaryBuffer(buffer, identifier);
}
}
}
{code}
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268463#comment-13268463 ]
Robert Joseph Evans commented on MAPREDUCE-4148:
------------------------------------------------
I like the patch a lot. It cleans things up which is really good. I think Daryn has a few minor comments but other then that I give it a +1. I would also like to try and pull this into branch-0.23. The patch seems to apply OK for branch-0.23, but if you do need any help with it I am happy to do it for you.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Attachment: MAPREDUCE-4148.patch
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270642#comment-13270642 ]
Daryn Sharp commented on MAPREDUCE-4148:
----------------------------------------
Looks like a simple fix: {{decodeIdentifier}} isn't checking for a null from {{getClassForIdentifier}} when it can't find the class.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270555#comment-13270555 ]
Robert Joseph Evans commented on MAPREDUCE-4148:
------------------------------------------------
I like the changes, but org.apache.hadoop.ipc.TestSaslRPC appears to be a real issue, where the test is not finding the needed token identifier.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273988#comment-13273988 ]
Hudson commented on MAPREDUCE-4148:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1041 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/])
MAPREDUCE-4148. MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199)
Result = FAILURE
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.0
>
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265934#comment-13265934 ]
Tom White commented on MAPREDUCE-4148:
--------------------------------------
Thanks for the reviews. I agree that using a ServiceLoader is the best approach for this one.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273334#comment-13273334 ]
Hudson commented on MAPREDUCE-4148:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2230 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2230/])
MAPREDUCE-4148. MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199)
Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.0
>
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265927#comment-13265927 ]
Robert Joseph Evans commented on MAPREDUCE-4148:
------------------------------------------------
I agree with Daryn on this. I don't really like using the static block to register things. We have had issues in the past in the RM where renewing delegation tokens did not work because the HDFS configuration files were not loaded in a static block because there was no reference to the class with the static block in the code being run. It was a beast to debug too.
I would prefer to see something that uses a ServiceLoader, or like Daryn said something from the secret manager could ensure that those static blocks are called for all of them that it cares about, but I think the service loader is the direction that a lot of the code has been going in.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269868#comment-13269868 ]
Daryn Sharp commented on MAPREDUCE-4148:
----------------------------------------
+1 Great job!
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268038#comment-13268038 ]
Hadoop QA commented on MAPREDUCE-4148:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12525539/MAPREDUCE-4148.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2354//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2354//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2354//console
This message is automatically generated.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272327#comment-13272327 ]
Daryn Sharp commented on MAPREDUCE-4148:
----------------------------------------
+1 Assuming that the test cited by Bobby also works now. Great job!
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273328#comment-13273328 ]
Hudson commented on MAPREDUCE-4148:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2304 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2304/])
MAPREDUCE-4148. MapReduce should not have a compile-time dependency on HDFS. (Revision 1337199)
Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337199
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/TestToken.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
* /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.0
>
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Attachment: MAPREDUCE-4148.patch
New patch addressing Daryn's feedback. I left in the Guava Maps usage since we already have that dependency.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-4148:
-----------------------------------
Fix Version/s: (was: 2.0.0)
2.0.1
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 2.0.1
>
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265553#comment-13265553 ]
Hadoop QA commented on MAPREDUCE-4148:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524325/MAPREDUCE-4148.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
-1 javadoc. The javadoc tool appears to have generated 16 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2337//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2337//console
This message is automatically generated.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273038#comment-13273038 ]
Hadoop QA commented on MAPREDUCE-4148:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526294/MAPREDUCE-4148.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2380//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2380//console
This message is automatically generated.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Attachment: MAPREDUCE-4148.patch
New patch to fix findbugs warnings.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265558#comment-13265558 ]
Hadoop QA commented on MAPREDUCE-4148:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524325/MAPREDUCE-4148.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
-1 javadoc. The javadoc tool appears to have generated 16 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2338//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2338//console
This message is automatically generated.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Attachment: MAPREDUCE-4148.patch
This patch removes the dependency by introducing a TokenIdentifierFactory so that TokenIdentifier instances can be created from their byte array representations, without the client having to know the TokenIdentifier subtype.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268631#comment-13268631 ]
Hadoop QA commented on MAPREDUCE-4148:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12525651/MAPREDUCE-4148.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2358//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2358//console
This message is automatically generated.
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a
compile-time dependency on HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4148:
---------------------------------
Attachment: MAPREDUCE-4148.patch
I've reworked the patch to use a service loader, and also to address Daryn's feedback. There is now a decodeIdentifier() method on Token, which is better than my original approach. The only thing I didn't change is the use of ReflectionUtils, since it's a convenient, centralized way to construct an object, and it handles the exceptions in a uniform way.
(I suspect test-patch will fail due to HADOOP-8354.)
> MapReduce should not have a compile-time dependency on HDFS
> -----------------------------------------------------------
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token debug information). We should remove this dependency and MapReduce's compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira