You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Daryn Sharp (JIRA)" <ji...@apache.org> on 2011/09/12 21:51:09 UTC
[jira] [Created] (HDFS-2326) hftp token renewal uses wrong port
hftp token renewal uses wrong port
----------------------------------
Key: HDFS-2326
URL: https://issues.apache.org/jira/browse/HDFS-2326
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.20.205.0
Reporter: Daryn Sharp
Priority: Critical
Fix For: 0.20.205.0
MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2326) hftp token renewal uses wrong port
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103717#comment-13103717 ]
Owen O'Malley commented on HDFS-2326:
-------------------------------------
I tried the case of using HftpFileSystem, both directly from the command line:
{quote}
% hadoop fs -ls hftp://nn:hp/my-dir
% hadoop fs -cat hftp://nn:hp/my-file
% hadoop distcp hftp://nn:hp/my-dir hdfs://nn:rp/copy-dir
{quote}
During the distcp job, I ensured that the JobTracker successfully renewed both the hdfs and hftp tokens.
I also tried hftp with a token that I acquired using fetchdt using both http and rpc.
{quote}
% hadoop fetchdt --webservice https://nn:shp my.tok
% export HADOOP_TOKEN_FILE_LOCATION my.tok
% hadoop fs -ls hftp://nn:hp/my-dir
% hadoop fs -cat hftp://nn:hp/my-file
% (repeat with rpc token)
{quote}
> hftp token renewal uses wrong port
> ----------------------------------
>
> Key: HDFS-2326
> URL: https://issues.apache.org/jira/browse/HDFS-2326
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Daryn Sharp
> Priority: Critical
> Fix For: 0.20.205.0
>
>
> MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2326) hftp token renewal uses wrong port
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103801#comment-13103801 ]
Daryn Sharp commented on HDFS-2326:
-----------------------------------
Oh, and did the {{FsShell}} commands run long enough to see the token renew? I suppose you'd have to drop the token expiration to a low value to force it to renew while catting a big file. It would probably be easier to write a test program that just instantiates {{HftpFileSystem}} and goes to sleep. Then watch the renewals occur.
> hftp token renewal uses wrong port
> ----------------------------------
>
> Key: HDFS-2326
> URL: https://issues.apache.org/jira/browse/HDFS-2326
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Daryn Sharp
> Priority: Critical
> Fix For: 0.20.205.0
>
>
> MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2326) hftp token renewal uses wrong port
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103686#comment-13103686 ]
Daryn Sharp commented on HDFS-2326:
-----------------------------------
I did not actually see/test for this possible issue. It's based on reading the code while ensuring that the changes are compatible with the ip->host change. The multiple code paths for the token's service manipulation is a bit tricky. Did you also test the cases where a {{HarFileSystem}} is directly instantiated, that it works when it acquires the token itself and schedules its own renewal?
> hftp token renewal uses wrong port
> ----------------------------------
>
> Key: HDFS-2326
> URL: https://issues.apache.org/jira/browse/HDFS-2326
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Daryn Sharp
> Priority: Critical
> Fix For: 0.20.205.0
>
>
> MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2326) hftp token renewal uses wrong port
Posted by "Owen O'Malley (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reassigned HDFS-2326:
-----------------------------------
Assignee: Owen O'Malley
> hftp token renewal uses wrong port
> ----------------------------------
>
> Key: HDFS-2326
> URL: https://issues.apache.org/jira/browse/HDFS-2326
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Daryn Sharp
> Assignee: Owen O'Malley
> Priority: Critical
> Fix For: 0.20.205.0
>
>
> MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2326) hftp token renewal uses wrong port
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103719#comment-13103719 ]
Owen O'Malley commented on HDFS-2326:
-------------------------------------
I also don't understand your reference to the HarFileSystem. Are you talking about har using an underlying hftp file system?
> hftp token renewal uses wrong port
> ----------------------------------
>
> Key: HDFS-2326
> URL: https://issues.apache.org/jira/browse/HDFS-2326
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Daryn Sharp
> Priority: Critical
> Fix For: 0.20.205.0
>
>
> MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2326) hftp token renewal uses wrong port
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103720#comment-13103720 ]
Owen O'Malley commented on HDFS-2326:
-------------------------------------
Oh, I think I see the source of the confusion. The Hftp file system copies the token and changes the *copy*.
> hftp token renewal uses wrong port
> ----------------------------------
>
> Key: HDFS-2326
> URL: https://issues.apache.org/jira/browse/HDFS-2326
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Daryn Sharp
> Priority: Critical
> Fix For: 0.20.205.0
>
>
> MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2326) hftp token renewal uses wrong port
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103099#comment-13103099 ]
Owen O'Malley commented on HDFS-2326:
-------------------------------------
I don't understand your description of the problem. I've tested getting, renewing and canceling delegation tokens for both hdfs and hftp. Can you provide more detail about what you tried and what happened?
> hftp token renewal uses wrong port
> ----------------------------------
>
> Key: HDFS-2326
> URL: https://issues.apache.org/jira/browse/HDFS-2326
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Daryn Sharp
> Priority: Critical
> Fix For: 0.20.205.0
>
>
> MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2326) hftp token renewal uses wrong port
Posted by "Owen O'Malley (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HDFS-2326.
---------------------------------
Resolution: Invalid
Fix Version/s: (was: 0.20.205.0)
> hftp token renewal uses wrong port
> ----------------------------------
>
> Key: HDFS-2326
> URL: https://issues.apache.org/jira/browse/HDFS-2326
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Daryn Sharp
> Assignee: Owen O'Malley
> Priority: Critical
>
> MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2326) hftp token renewal uses wrong port
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103786#comment-13103786 ]
Daryn Sharp commented on HDFS-2326:
-----------------------------------
(Ack, I was reviewing har changes this morning. I meant hftp)
The use case is: Hftp finds an hdfs token. The service is the rpc port. The token is passed to {{setDelegationToken}}. {{renewToken}} is set to the original token, which contains the rpc port. Then a copy of the token is made with the kind/service squished to hdfs. Hftp's renewer will use the unchanged token in {{renewToken}}, which contains the rpc port instead of the http port.
> hftp token renewal uses wrong port
> ----------------------------------
>
> Key: HDFS-2326
> URL: https://issues.apache.org/jira/browse/HDFS-2326
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Daryn Sharp
> Priority: Critical
> Fix For: 0.20.205.0
>
>
> MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2326) hftp token renewal uses wrong port
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105632#comment-13105632 ]
Owen O'Malley commented on HDFS-2326:
-------------------------------------
As long as the kind and service are set together, it works. If the original token is a hdfs delegation token and the service is the rpc port, the renewal will use the rpc renewal.
> hftp token renewal uses wrong port
> ----------------------------------
>
> Key: HDFS-2326
> URL: https://issues.apache.org/jira/browse/HDFS-2326
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.205.0
> Reporter: Daryn Sharp
> Priority: Critical
> Fix For: 0.20.205.0
>
>
> MAPREDUCE-2764 introduced a new method for token renewal. The change appears to have a problem renewing hftp tokens. The {{setDelegationToken}} method will reset the token's service to contain the remote rpc port. However, the renewer expects it to be the remote https port.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira