You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Daryn Sharp (Created) (JIRA)" <ji...@apache.org> on 2011/11/23 20:25:40 UTC
[jira] [Created] (HDFS-2589) unnecessary hftp token fetch and
renewal thread
unnecessary hftp token fetch and renewal thread
-----------------------------------------------
Key: HDFS-2589
URL: https://issues.apache.org/jira/browse/HDFS-2589
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 0.20.205.1
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread. With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running. This thread should not be running in the JT.
After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and
renewal thread
Posted by "Daryn Sharp (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158899#comment-13158899 ]
Daryn Sharp commented on HDFS-2589:
-----------------------------------
I 100% agree the error handling is less than ideal, but I don't believe I fundamentally changed the behavior. The pre-existing code already assumed that any exception means the other side is insecure. I only made it defer the fetching of the token.
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
> Key: HDFS-2589
> URL: https://issues.apache.org/jira/browse/HDFS-2589
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread. With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running. This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and
renewal thread
Posted by "Jitendra Nath Pandey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168657#comment-13168657 ]
Jitendra Nath Pandey commented on HDFS-2589:
--------------------------------------------
+1.
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
> Key: HDFS-2589
> URL: https://issues.apache.org/jira/browse/HDFS-2589
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread. With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running. This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and
renewal thread
Posted by "Tsz Wo (Nicholas), SZE (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168960#comment-13168960 ]
Tsz Wo (Nicholas), SZE commented on HDFS-2589:
----------------------------------------------
Hi Daryn,
It seems that it is better to change HftpFileSystem but not DelegationTokenRenewer.
In WebHdfsFileSystem, the lazy initialization is implemented in addRenewAction(..), which is the only method using DT_RENEWER. It is more lazy -- we even don't have to create a DelegationTokenRenewer object if addRenewAction(..) is not invoked.
{code}
//WebHdfsFileSystem
private static DelegationTokenRenewer<WebHdfsFileSystem> DT_RENEWER = null;
private static synchronized void addRenewAction(final WebHdfsFileSystem webhdfs) {
if (DT_RENEWER == null) {
DT_RENEWER = new DelegationTokenRenewer<WebHdfsFileSystem>(WebHdfsFileSystem.class);
DT_RENEWER.start();
}
DT_RENEWER.addRenewAction(webhdfs);
}
{code}
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
> Key: HDFS-2589
> URL: https://issues.apache.org/jira/browse/HDFS-2589
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread. With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running. This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2589) unnecessary hftp token fetch and
renewal thread
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HDFS-2589:
------------------------------
Status: Patch Available (was: Open)
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
> Key: HDFS-2589
> URL: https://issues.apache.org/jira/browse/HDFS-2589
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 0.20.205.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread. With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running. This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2589) unnecessary hftp token fetch and
renewal thread
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-2589:
-----------------------------
Target Version/s: 1.1.0 (was: 1.0.0)
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
> Key: HDFS-2589
> URL: https://issues.apache.org/jira/browse/HDFS-2589
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread. With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running. This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2589) unnecessary hftp token fetch and
renewal thread
Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HDFS-2589:
------------------------------
Attachment: HDFS-2589.patch
implements lazy fetch and thread
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
> Key: HDFS-2589
> URL: https://issues.apache.org/jira/browse/HDFS-2589
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 0.20.205.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread. With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running. This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2589) unnecessary hftp token fetch and
renewal thread
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-2589:
-----------------------------
Resolution: Fixed
Fix Version/s: 1.0.0
Target Version/s: 1.0.0 (was: 1.1.0)
Status: Resolved (was: Patch Available)
Committed to 1.0.0 and branch-1 on the strength of Jitendra's +1. Feel free to improve in 1.1.0.
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
> Key: HDFS-2589
> URL: https://issues.apache.org/jira/browse/HDFS-2589
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Fix For: 1.0.0
>
> Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread. With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running. This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and
renewal thread
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156217#comment-13156217 ]
Hadoop QA commented on HDFS-2589:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504914/HDFS-2589.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1607//console
This message is automatically generated.
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
> Key: HDFS-2589
> URL: https://issues.apache.org/jira/browse/HDFS-2589
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 0.20.205.1
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread. With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running. This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and
renewal thread
Posted by "Jitendra Nath Pandey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158875#comment-13158875 ]
Jitendra Nath Pandey commented on HDFS-2589:
--------------------------------------------
I am not too sure about remoteIsInsecure check, because it will be set to true for any kind of exception.
The patch looks ok to me, otherwise.
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
> Key: HDFS-2589
> URL: https://issues.apache.org/jira/browse/HDFS-2589
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 1.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread. With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running. This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira