You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Siddharth Seth (Created) (JIRA)" <ji...@apache.org> on 2011/12/10 04:46:39 UTC
[jira] [Created] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
TokenCache does not cache viewfs credentials correctly
------------------------------------------------------
Key: MAPREDUCE-3529
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Critical
viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
{noformat}
Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
at org.apache.hadoop.ipc.Client.call(Client.java:1085)
at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
at $Proxy8.getDelegationToken(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
at $Proxy8.getDelegationToken(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
{noformat}
This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3529:
--------------------------------------
Status: Patch Available (was: Open)
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3529:
--------------------------------------
Attachment: MR3529_v1.txt
Tested along with the patch posted on HDFS-2665. Still needs to be tested for the actual pig issue(s) though.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3529:
--------------------------------------
Attachment: MR3529_v2.txt
Patch with unit tests. Also includes changes and tests for MAPREDUCE-3455.
Waiting for HDFS-2655 before submitting.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Daryn Sharp (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179847#comment-13179847 ]
Daryn Sharp commented on MAPREDUCE-3529:
----------------------------------------
+1 It'd be nice if the default impl of {{FileSystem#getDelegationTokens}} returned a list based on a call to {{getDelegationToken}} so that the {{TokenCache}} didn't have to call both. I guess that can be another jira.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3529:
--------------------------------------
Status: Open (was: Patch Available)
Cancelling - Jenkins doesn't seem to have picked up the HADOOP-7933.
Thanks for taking a look Daryn,
bq. If fs.getDelegationTokens(delegTokenRenewer, credentials) returns null because all tokens are already acquired, isn't this going to cause fs.getDelegationTokens(delegTokenRenewer) to be unnecessarily called?
fs.getDelegationTokens(renewer, creds) is supposed to return the full list of credentials (not an incremental list). If there's no tokens - the assumption is the API may not have been implemented and hence the fallback to getDelegationToken.
bq. credentials.addToken(fsNameText, token) really should be credentials.addToken(token.getService(), token).
Will upload a new patch with this change.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179969#comment-13179969 ]
Hudson commented on MAPREDUCE-3529:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1494 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1494/])
MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly. (sseth)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227374
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3529:
--------------------------------------
Status: Open (was: Patch Available)
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180363#comment-13180363 ]
Hudson commented on MAPREDUCE-3529:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #151 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/151/])
merge MAPREDUCE-3529 from trunk
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227375
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3529:
--------------------------------------
Attachment: MR3529_v3.txt
Updated patch based on review feedback.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179949#comment-13179949 ]
Arun C Murthy commented on MAPREDUCE-3529:
------------------------------------------
+1 lgtm.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180332#comment-13180332 ]
Hudson commented on MAPREDUCE-3529:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #129 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/129/])
merge MAPREDUCE-3529 from trunk
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227375
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179992#comment-13179992 ]
Hudson commented on MAPREDUCE-3529:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #348 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/348/])
merge MAPREDUCE-3529 from trunk
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227375
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3529:
--------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.1
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the reviews Daryn and Arun. Committed to trunk and branch-0.23.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180320#comment-13180320 ]
Hudson commented on MAPREDUCE-3529:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #916 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/916/])
MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly. (sseth)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227374
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3529:
--------------------------------------
Attachment: MR3529_v4.txt
Added apache header to the unit test.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172969#comment-13172969 ]
Hadoop QA commented on MAPREDUCE-3529:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12508048/MR3529_v1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed the unit tests build
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1478//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1478//console
This message is automatically generated.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180377#comment-13180377 ]
Hudson commented on MAPREDUCE-3529:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #949 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/949/])
MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly. (sseth)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227374
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179999#comment-13179999 ]
Hudson commented on MAPREDUCE-3529:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1514 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1514/])
MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly. (sseth)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227374
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3529:
--------------------------------------
Status: Patch Available (was: Open)
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179968#comment-13179968 ]
Hudson commented on MAPREDUCE-3529:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1566 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1566/])
MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly. (sseth)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227374
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179974#comment-13179974 ]
Hudson commented on MAPREDUCE-3529:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #337 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/337/])
merge MAPREDUCE-3529 from trunk
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227375
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179816#comment-13179816 ]
Hadoop QA commented on MAPREDUCE-3529:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12509336/MR3529_v4.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1537//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1537//console
This message is automatically generated.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178962#comment-13178962 ]
Hadoop QA commented on MAPREDUCE-3529:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12508124/MR3529_v2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed the unit tests build
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1527//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1527//console
This message is automatically generated.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Daryn Sharp (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178952#comment-13178952 ]
Daryn Sharp commented on MAPREDUCE-3529:
----------------------------------------
If {{fs.getDelegationTokens(delegTokenRenewer, credentials)}} returns null because all tokens are already acquired, isn't this going to cause {{fs.getDelegationTokens(delegTokenRenewer)}} to be unnecessarily called?
{{credentials.addToken(fsNameText, token)}} really should be {{credentials.addToken(token.getService(), token)}}.
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3529) TokenCache does not cache viewfs
credentials correctly
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3529:
--------------------------------------
Status: Patch Available (was: Open)
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3529_v1.txt, MR3529_v2.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3529) TokenCache does not cache
viewfs credentials correctly
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179967#comment-13179967 ]
Hudson commented on MAPREDUCE-3529:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #326 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/326/])
merge MAPREDUCE-3529 from trunk
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227375
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java
> TokenCache does not cache viewfs credentials correctly
> ------------------------------------------------------
>
> Key: MAPREDUCE-3529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3529
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3529_v1.txt, MR3529_v2.txt, MR3529_v3.txt, MR3529_v4.txt
>
>
> viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
> Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
> {noformat}
> Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4027)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:281)
> at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
> at $Proxy8.getDelegationToken(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:100)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:65)
> at $Proxy8.getDelegationToken(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:456)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:812)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationTokens(DistributedFileSystem.java:839)
> at org.apache.hadoop.fs.viewfs.ChRootedFileSystem.getDelegationTokens(ChRootedFileSystem.java:311)
> at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDelegationTokens(ViewFileSystem.java:490)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:144)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:91)
> at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:84)
> {noformat}
> This will likely require some changes in viewfs/hdfs - will open a Jira with details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira