You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Daryn Sharp (Created) (JIRA)" <ji...@apache.org> on 2011/11/23 20:25:40 UTC

[jira] [Created] (HDFS-2589) unnecessary hftp token fetch and renewal thread

unnecessary hftp token fetch and renewal thread
-----------------------------------------------

                 Key: HDFS-2589
                 URL: https://issues.apache.org/jira/browse/HDFS-2589
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: security
    Affects Versions: 0.20.205.1
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread.  With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running.  This thread should not be running in the JT.

After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and renewal thread

Posted by "Daryn Sharp (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158899#comment-13158899 ] 

Daryn Sharp commented on HDFS-2589:
-----------------------------------

I 100% agree the error handling is less than ideal, but I don't believe I fundamentally changed the behavior.  The pre-existing code already assumed that any exception means the other side is insecure.  I only made it defer the fetching of the token.
                
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
>                 Key: HDFS-2589
>                 URL: https://issues.apache.org/jira/browse/HDFS-2589
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 1.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread.  With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running.  This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and renewal thread

Posted by "Jitendra Nath Pandey (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168657#comment-13168657 ] 

Jitendra Nath Pandey commented on HDFS-2589:
--------------------------------------------

+1.
                
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
>                 Key: HDFS-2589
>                 URL: https://issues.apache.org/jira/browse/HDFS-2589
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 1.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread.  With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running.  This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and renewal thread

Posted by "Tsz Wo (Nicholas), SZE (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168960#comment-13168960 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-2589:
----------------------------------------------

Hi Daryn, 

It seems that it is better to change HftpFileSystem but not DelegationTokenRenewer.

In WebHdfsFileSystem, the lazy initialization is implemented in addRenewAction(..), which is the only method using DT_RENEWER.  It is more lazy -- we even don't have to create a DelegationTokenRenewer object if addRenewAction(..) is not invoked.
{code}
//WebHdfsFileSystem
  private static DelegationTokenRenewer<WebHdfsFileSystem> DT_RENEWER = null;

  private static synchronized void addRenewAction(final WebHdfsFileSystem webhdfs) {
    if (DT_RENEWER == null) {
      DT_RENEWER = new DelegationTokenRenewer<WebHdfsFileSystem>(WebHdfsFileSystem.class);
      DT_RENEWER.start();
    }

    DT_RENEWER.addRenewAction(webhdfs);
  }
{code}
                
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
>                 Key: HDFS-2589
>                 URL: https://issues.apache.org/jira/browse/HDFS-2589
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 1.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread.  With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running.  This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-2589) unnecessary hftp token fetch and renewal thread

Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HDFS-2589:
------------------------------

    Status: Patch Available  (was: Open)
    
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
>                 Key: HDFS-2589
>                 URL: https://issues.apache.org/jira/browse/HDFS-2589
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.20.205.1
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread.  With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running.  This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-2589) unnecessary hftp token fetch and renewal thread

Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HDFS-2589:
-----------------------------

    Target Version/s: 1.1.0  (was: 1.0.0)
    
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
>                 Key: HDFS-2589
>                 URL: https://issues.apache.org/jira/browse/HDFS-2589
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 1.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread.  With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running.  This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-2589) unnecessary hftp token fetch and renewal thread

Posted by "Daryn Sharp (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HDFS-2589:
------------------------------

    Attachment: HDFS-2589.patch

implements lazy fetch and thread
                
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
>                 Key: HDFS-2589
>                 URL: https://issues.apache.org/jira/browse/HDFS-2589
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.20.205.1
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread.  With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running.  This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-2589) unnecessary hftp token fetch and renewal thread

Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HDFS-2589:
-----------------------------

          Resolution: Fixed
       Fix Version/s: 1.0.0
    Target Version/s: 1.0.0  (was: 1.1.0)
              Status: Resolved  (was: Patch Available)

Committed to 1.0.0 and branch-1 on the strength of Jitendra's +1.  Feel free to improve in 1.1.0.
                
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
>                 Key: HDFS-2589
>                 URL: https://issues.apache.org/jira/browse/HDFS-2589
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 1.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 1.0.0
>
>         Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread.  With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running.  This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and renewal thread

Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156217#comment-13156217 ] 

Hadoop QA commented on HDFS-2589:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12504914/HDFS-2589.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1607//console

This message is automatically generated.
                
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
>                 Key: HDFS-2589
>                 URL: https://issues.apache.org/jira/browse/HDFS-2589
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.20.205.1
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread.  With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running.  This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and renewal thread

Posted by "Jitendra Nath Pandey (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158875#comment-13158875 ] 

Jitendra Nath Pandey commented on HDFS-2589:
--------------------------------------------

I am not too sure about remoteIsInsecure check, because it will be set to true for any kind of exception.
The patch looks ok to me, otherwise.
                
> unnecessary hftp token fetch and renewal thread
> -----------------------------------------------
>
>                 Key: HDFS-2589
>                 URL: https://issues.apache.org/jira/browse/HDFS-2589
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 1.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-2589.patch
>
>
> Instantiation of the hftp filesystem is causing a token to be implicitly created and added to a custom token renewal thread.  With the new token renewal feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled token, and an explicit token) and leave a spurious renewal thread running.  This thread should not be running in the JT.
> After speaking with Owen, the quick solution is to lazy fetch the token, and to lazy start the renewer thread.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira