You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Daryn Sharp (JIRA)" <ji...@apache.org> on 2012/09/07 19:04:08 UTC
[jira] [Created] (HADOOP-8779) Use tokens regardless of
authentication type
Daryn Sharp created HADOOP-8779:
-----------------------------------
Summary: Use tokens regardless of authentication type
Key: HADOOP-8779
URL: https://issues.apache.org/jira/browse/HADOOP-8779
Project: Hadoop Common
Issue Type: New Feature
Components: fs, security
Affects Versions: 3.0.0, 2.2.0-alpha
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482763#comment-13482763 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
bq. That's not how the token routines are conditionalized today. Some simply return null if isSecurityEnabled is false.
Today, tokens are only issued when Kerberos is used and using Kerberos is synonymous to security being turned on. Hence isSecurityEnabled is used as a proxy for checking if Kerberos is used. When Kerberos is not the only initial auth method to be paired with tokens, the checking should be "is the client authenticated using the configured initial auth method(s)"?
bq. I don't believe it's the filesystem's responsibility to decide if a token can be issued.
The token is called NN delegation token. It's a credential that NN generates and manages for its clients to connect back. It is the sole responsibility of NN to decide whether it should issue, expire, or validate/accept a token for a given client. You lost me here. :-)
bq. If the user/job-client requests a token, then it should try to issue one.
Why? If SIMPLE instead of TOKEN is configured as subsequent auth method, why issue a token that will never be used? Simplifying code is good, but not to the extend that unnecessary objects are created and exchanged at runtime.
bq. We are not in stark opposition on this point. I'd rather we don't have multiple code paths, but we can add conditionals to the job client to enable/disable token fetching, and to the RPC client to only use tokens to allow SIMPLE + SIMPLE.
I don't think adding conditionals to the job client is needed. A config option for subsequent auth method should suffice for now. This option decides whether the job client should fetch tokens and whether RPC client for jobs should use tokens. It also tells NN whether it should issue tokens.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484127#comment-13484127 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
bq. Even if we used SASL PLAIN, we would still have to differentiate between PLAIN and DIGEST-MD5, so that NN knows when to start its SecretManager. In particular, when PLAIN is configured, it shouldn't trigger the isSecurityEnabled code path.
I'm making {{isSecurityEnabled}} mean SASL is being used. SIMPLE doesn't change at all and doesn't use tokens, while PLAIN means security is enabled and tokens are required for internal auth.
bq. (Re: Multiple internal auths) We need to support at least 2, SIMPLE and TOKEN. I've said that repeatedly, I hope this time it gets to you. :)
There's no debate, we already are and have in completely agreement. In these jiras you've hinted at selectable internal auths, so all I meant to clarify is security (!SIMPLE) for internal auth is token, SIMPLE is SIMPLE which is why I've chosen the PLAIN route.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478331#comment-13478331 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
On the server side, you can support both SIMPLE and DIEGST-MD5 (token) simultaneously. It's up to the client to choose which method to use. But the client side code needs to make a decision. Without an option to specify the desired behavior, the client code can only default to one behavior. See more discussion in HDFS-4056.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477497#comment-13477497 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
I agree the use of tokens for subsequent authentication (referred to as internal auth in previous discussions, but maybe subsequent auth is a better name?) shouldn't be limited to Kerberos authenticated initial connections (referred to as external auth in previous discussions, but maybe initial auth is better name?). However, IMHO, we should give users the option not to use tokens for subsequent authentication, as is the case when security is turned off today. See HDFS-4056 for more discussion. https://issues.apache.org/jira/browse/HDFS-4056?focusedCommentId=13477142&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13477142
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451135#comment-13451135 ]
Aaron T. Myers commented on HADOOP-8779:
----------------------------------------
I think it's a bit of a misrepresentation to say that tokens are only used for authorization. Yes, they also carry information that can be used to authorize an otherwise authenticated client, but in some cases they are also used for authentication itself, e.g. in the case of BlockTokens.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450887#comment-13450887 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
bq. Authorization without authentication or, in other words, allowing access to resources to users without knowing who they really are - doesn't make much sense. But if you call it as "simple auth" instead of "security off", it becomes meaningful. Arguably I am just debating semantics here.
No debate, I fully agree it should be considered "simple auth". I used 'security "off"' because it's a misnomer. I look at it as going to a private party and need a key (token) to open the door (service). I can either tell the guard who I am and he trusts me (simple auth) or I can present a secure picture id (kerberos). However I get the key (token), I'm only authorized to unlock certain doors (services) after that point.
bq. To address your needs, you can use DIGEST auth-method?
Yes. I'm allowing a server configured for simple auth to:
# Grant tokens if requested - server currently will not
# Validate tokens if presented via sasl DIGEST - sever currently kicks the client out of sasl into simple auth
# Continue to allow a simple auth client to only use simple auth in order to preserve backwards compat with older clients
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481954#comment-13481954 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
bq. 3. Remove all the conditionals from the filesystems for whether tokens can be acquired and/or used
I'm not sure this should be done, since the filesystem still needs to decide whether a token can be issued based on whether the client is authenticated using the configured initial auth method.
bq. 4. Always submit jobs with tokens
I think this is where we have differences. In my view, when SIMPLE + SIMPLE is configured, there should be no tokens issued or submitted (as it is today).
bq. you formerly agreed this was worthwhile: "Robert Joseph Evans and Daryn Sharp, I agree we should allow SIMPLE auth to be coupled with tokens." Has your position changed?
No. We should allow SIMPLE to be paired with tokens. But that doesn't mean SIMPLE should always be paired with tokens. In particular, when SIMPLE is paired with SIMPLE on a given installation (configuration), we want to avoid the overhead of using tokens.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479171#comment-13479171 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
I think the "AuthMethod" field in the Connection class should be decided primarily on 1) connection type (initial or subsequent) and 2) what auth method is configured for that connection type. What credentials the client currently has only comes into play when multiple auth methods are configured for the intended connection type and we have a choice.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478377#comment-13478377 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
The RPC client will use DIGEST-MD5 if and only if there's a token in the UGI. The rpc client used to only look for tokens if auth is kerberos, but HADOOP-8784 changed the client to always look for tokens regardless of auth method. I believe your real question, which is legitimate, is whether the job client chooses to get fs tokens. We can discuss that on this jira because it's unrelated to HDFS-4056.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456453#comment-13456453 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
Actually HADOOP-8758 is supposed to be a small patch and the meat of it is this decoupling.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486088#comment-13486088 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
bq. This might work. isSecurityEnabled is extensively used in the code base. Pls make sure the new semantics fits in all cases (especially where it might be used to mean "Kerberos is enabled").
Yes, this is exactly what I'm doing. :)
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483877#comment-13483877 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
bq. I think we're starting to over-engineer the issue at hand and bleeding requirements for other jiras like HADOOP-8758 into the design.
I don't feel that way. I would have made the same comments and suggestions even if our goal was to simply enable SIMPLE + TOKEN without the perspective of adding any new auth method (ex., TOKEN + TOKEN, which is the goal of HADOOP-8758) in the future. To pair tokens with more than one initial auth method isn't trivial. Otherwise, we would have done it in the first implementation of tokens. As I said earlier, the main work of HADOOP-8758 is actually about removing the tight coupling of Kerberos with tokens, and not about adding new auth methods. But you wanted to do it in this JIRA. Hope you didn't regret it. :-)
bq. I feel we have become excessively bogged down trying to supporting SIMPLE with tokens (my initial goal) and w/o tokens (your goal).
SIMPLE w/o tokens is currently supported and I think there is value in keep supporting it. I'm sorry if this work turns out to be more than what you planned, but we should try to enable/optimize Hadoop for more use cases rather than less, right?
bq. A far simpler approach is supporting SASL's PLAIN mechanism. It's basically hadoop's SIMPLE, but within the context of SASL. The end result is that PLAIN will trigger all the isSecurityEnabled code w/o changing any of the behavior of SIMPLE. The change becomes almost trivial because I won't have to touch the security conditionals in filesystems, mr, or yarn.
The reason why we didn't use SASL's PLAIN mechanism is we didn't want to pay the overhead of SASL when security is turned off. But it is irrelevant here. Even if we used SASL PLAIN, we would still have to differentiate between PLAIN and DIGEST-MD5, so that NN knows when to start its SecretManager. In particular, when PLAIN is configured, it shouldn't trigger the isSecurityEnabled code path.
bq. I don't feel we need to ponder multiple internal auths
We need to support at least 2, SIMPLE and TOKEN. I've said that repeatedly, I hope this time it gets to you. :-)
Btw, I don't have bandwidth to respond to you on a daily basis (as I have done in the past few days). I apologize if my responses appear to be a little slow. And I wasn't able to provide comments on subtasks HADOOP-8783 and HADOOP-8784 before they were committed. They escaped my radar since their titles didn't sound like they were related to this JIRA. I thought they were mere "improvements", not behavior changing patches. :-)
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451502#comment-13451502 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
Firstly, delegation tokens are not authorizations; they are just credentials to be used with some authentication method (DIGEST-MD5 in this case). There could be many ways to set up the credentials to be used with DIGEST-MD5. Hence, we have many different delegation token implementations (SecretManagers) in Hadoop. But currently only a single type of delegation tokens (to be used with DIGEST-MD5 as the internal auth method) can be used for a particular service and it is tightly coupled with Kerberos as the only external auth method. HADOOP-8758 is opened to support DIGEST-MD5 as an external auth method (with potentially many different types of tokens to be used with it). See my comment in HADOOP-8758 for explanation on external vs. internal auth methods.
Secondly, this JIRA is not needed; it is already assumed by HADOOP-8758. The decoupling of Kerberos from existing delegation token implementations (used with DIGEST-MD5 as internal auth method) has to be done before adding DIGEST-MD5 as an external auth method. Once decoupling is done, auth methods other than DIGEST-MD5 (including SIMPLE auth) should also be configurable as external auth methods.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485527#comment-13485527 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
bq. Even if we used SASL PLAIN, we would still have to differentiate between PLAIN and DIGEST-MD5, so that NN knows when to start its SecretManager.
Sorry, my above comment should read "... differentiate between PLAIN and Kerberos, ...", since I was talking about initial auth methods.
bq. I'm making isSecurityEnabled mean SASL is being used. SIMPLE doesn't change at all and doesn't use tokens, while PLAIN means security is enabled and tokens are required for internal auth.
This might work. isSecurityEnabled is extensively used in the code base. Pls make sure the new semantics fits in all cases (especially where it might be used to mean "Kerberos is enabled").
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482330#comment-13482330 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
bq. {quote}3. Remove all the conditionals from the filesystems for whether tokens can be acquired and/or used{quote}
bq. I'm not sure this should be done, since the filesystem still needs to decide whether a token can be issued based on whether the client is authenticated using the configured initial auth method.
That's not how the token routines are conditionalized today. Some simply return null if {{isSecurityEnabled}} is false.
I don't believe it's the filesystem's responsibility to decide if a token can be issued. If the user/job-client requests a token, then it should try to issue one. This will help solve incompatibilities where secure & insecure clusters cannot be accessed with the same client config.
bq. I think this is where we have differences. In my view, when SIMPLE + SIMPLE is configured, there should be no tokens issued or submitted (as it is today).
We are not in stark opposition on this point. I'd rather we don't have multiple code paths, but we can add conditionals to the job client to enable/disable token fetching, and to the RPC client to only use tokens to allow SIMPLE + SIMPLE.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450794#comment-13450794 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
The security framework currently only uses tokens for authz if and only if kerberos is the auth method. With security "off", this introduces multiple code paths with added complexity and reduced code coverage. Pre-commit integration tests cannot verify that authz/tokens work, so devs w/o access to a secure cluster may unexpectedly break tokens.
The change to allow tokens with simple auth must be backwards compatible.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483342#comment-13483342 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
I think we're starting to over-engineer the issue at hand and bleeding requirements for other jiras like HADOOP-8758 into the design. This jira has a different goal that happens to be a step in that direction, but this jira has a much simpler goal: supporting tokens on "insecure" clusters. I had hoped to eliminate all the conditional code paths for security and make SIMPLE behave no differently than other auth methods. I feel we have become excessively bogged down trying to supporting SIMPLE with tokens (my initial goal) and w/o tokens (your goal).
A far simpler approach is supporting SASL's PLAIN mechanism. It's basically hadoop's SIMPLE, but within the context of SASL. The end result is that PLAIN will trigger all the {{isSecurityEnabled}} code w/o changing any of the behavior of SIMPLE. The change becomes almost trivial because I won't have to touch the security conditionals in filesystems, mr, or yarn.
Then all we need is a conf key, such as the one in HADOOP-8965, to force clients (ie. a task) to use a token instead of attempting an auth. I don't feel we need to ponder multiple internal auths - tokens can be used with any external auth as this approach will demonstrate. Users will get a clean error message from failed tasks such as "hey, you don't have a token!" instead of huge cryptic SASL exceptions. I'd like to keep this change succinct and leave broader changes to other jiras.
Is this approach more palatable? The only downside is that we still need to do dual testing for secure/insecure, which I'd hoped to eliminate... But at least the pre-commits finally have the option to use PLAIN auth to test security.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452054#comment-13452054 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
I don't mind what jargon we use. We'll call tokens another auth method.
bq. Secondly, this JIRA is not needed; it is already assumed by HADOOP-8758. The decoupling of Kerberos from existing delegation token implementations [...]
The intention of this jira is to break the artificial limitation that the client will not request tokens if auth is simple, and the server will refuse tokens if the auth is simple.
I split this off because it's a much simpler high value change that will prevent HADOOP-8785 from being a big-bang patch. If you think this should be a sub-task, I think that's reasonable.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481914#comment-13481914 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
bq. {quote} I've long considered whether job submission should set a conf key that forces a task to only use tokens which is what I think you are also suggesting.{quote}
bq. For example, the job client upper layer could decide on which auth method to use and tell RPC client to use that, or it could just pass to the RPC client the connection type (initial or subsequent) and let RPC client choose the right auth method.
I think we might be in violent agreement...?
bq. I think we should agree on the overall design and what behaviors are supported at each piece, at least at a high-level, before making changes to any of them
The design is rather simple at a high level. I'll write up a doc if this isn't clear, or a good basis for discussion:
# Allow tokens to be issued for any non-token authentication (SIMPLE/KERBEROS/etc) method
# Allow clients to use said tokens for subsequent connections
# Remove all the conditionals from the filesystems for whether tokens can be acquired and/or used
# Always submit jobs with tokens
#* maybe controlled via a conf setting, would prefer no setting to reduce code complexity
#* add a conf key to the job conf that instructs the task's RPC client that it must use tokens
Everything is already in place within the RPC layer to support tokens with any auth if a secret manager is enabled. Hence, HDFS-4056 is attempting to activate the secret manager which places us between steps 2 & 3.
Removing the conditionals in the filesystems doesn't mandate tokens with SIMPLE auth, but allows them to be used if the job client requests them. Step 4 is where we decide when and if tokens are required.
bq. For example, if we agree on supporting SIMPLE + SIMPLE, HDFS-4056 is not needed.
In https://issues.apache.org/jira/browse/HADOOP-8758?focusedCommentId=13448518&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13448518, you formerly agreed this was worthwhile: "Robert Joseph Evans and Daryn Sharp, I agree we should allow SIMPLE auth to be coupled with tokens." Has your position changed?
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479272#comment-13479272 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
There could be connections that can be of either type (don't care), in which case any legal auth method can be used. When multiple auth methods (or multiple credentials for the same auth method, e.g., multiple types of tokens) may be used, how to support automatic failover from one method/credential to another is another thing to consider.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482155#comment-13482155 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
Just to clarify, IMO, a cluster may be configured to use SIMPLE + SIMPLE or SIMPLE + TOKEN, but not both at the same time. I don't see a use case for both at the same time. On the other hand, I do see the value of SIMPLE + SIMPLE without the overhead of tokens (i.e., without simultaneously enabling SIMPLE + TOKEN).
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478488#comment-13478488 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
Well, the job client scenario is just an example. The deeper issue is whether the client should make its decision on what auth method to use based on configuration, or on what credentials are currently available. I think the former is better and easier to reason. If the required credentials are not available, it should complain rather than automatically switch to make a different type of connection (a task switching from token to SIMPLE would defeat your testing purpose). Using credential precedence may work in simpler scenarios (like we had before), but may not work well when more combinations of initial and subsequent auth methods need to be supported.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478008#comment-13478008 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
I believe you are misunderstanding the patch, or I have an unintentional bug. To save people a trip to HDFS-4056, here's the relevant info:
bq. SIMPLE is still supported as it currently is today. The difference is that a SIMPLE auth connection may also use a token, whereas today an insecure cluster will kick a client back into SIMPLE auth if it attempts to negotiate DIGEST-MD5 with a token. Clients are not strictly required to use a token.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450834#comment-13450834 ]
Vinod Kumar Vavilapalli commented on HADOOP-8779:
-------------------------------------------------
Authorization without authentication or, in other words, allowing access to resources to users without knowing who they really are - doesn't make much sense. But if you call it as "simple auth" instead of "security off", it becomes meaningful. Arguably I am just debating semantics here.
bq. The security framework currently only uses tokens for authz if and only if kerberos is the auth method.
Kerberos and digest? To address your needs, you can use DIGEST auth-method?
That said, I agree that decoupling them will help a bit. In MR code, for e.g., we always maintained a separate authorization flag for queue and job level ACLs.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.2.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481489#comment-13481489 ]
Daryn Sharp commented on HADOOP-8779:
-------------------------------------
bq. The deeper issue is whether the client should make its decision on what auth method to use based on configuration, or on what credentials are currently available. I think the former is better and easier to reason.
I think we need to be clear on which client we are discussing to avoid confusion. The is the low-level RCP client uses a token if available, else kerberos or simple. Then there's a high-level client, like the job client, that needs to determine if it should get a token.
bq. If the required credentials are not available, it should complain rather than automatically switch to make a different type of connection (a task switching from token to SIMPLE would defeat your testing purpose)
True, it would defeat the purpose, which is why I've long considered whether job submission should set a conf key that forces a task to only use tokens which is what I think you are also suggesting. This would help with secure clusters to prevent the user from seeing a large confusing sasl exception from the rpc client when a token is unavailable. I planned to raise this issue when we get to MR's job client deciding if it get should get tokens.
I agree these are all very valid questions that we need to address. I hope these don't block HDFS-4056 and HADOOP-8785 (not posted because it depends on HDFS-4056). These jiras are incremental steps forward that are independent from this larger discussion. These jiras will not change job submission or task execution behavior until the job client is changed.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8779) Use tokens regardless of
authentication type
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481771#comment-13481771 ]
Kan Zhang commented on HADOOP-8779:
-----------------------------------
bq. I think we need to be clear on which client we are discussing to avoid confusion. The is the low-level RCP client uses a token if available, else kerberos or simple. Then there's a high-level client, like the job client, that needs to determine if it should get a token.
I'd consider that implementation detail. Once we agree on how the client should behave, we can then look at where it should be implemented. For example, the job client upper layer could decide on which auth method to use and tell RPC client to use that, or it could just pass to the RPC client the connection type (initial or subsequent) and let RPC client choose the right auth method. I'd prefer the latter since it would make it easier to support automatic failover from one auth method to another down the road.
bq. I agree these are all very valid questions that we need to address. I hope these don't block HDFS-4056 and HADOOP-8785 (not posted because it depends on HDFS-4056). These jiras are incremental steps forward that are independent from this larger discussion. These jiras will not change job submission or task execution behavior until the job client is changed.
I think we should agree on the overall design and what behaviors are supported at each piece, at least at a high-level, before making changes to any of them. For example, if we agree on supporting SIMPLE + SIMPLE, HDFS-4056 is not needed.
> Use tokens regardless of authentication type
> --------------------------------------------
>
> Key: HADOOP-8779
> URL: https://issues.apache.org/jira/browse/HADOOP-8779
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, security
> Affects Versions: 3.0.0, 2.0.2-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
>
> Security is a combination of authentication and authorization (tokens). Authorization may be granted independently of the authentication model. Tokens should be used regardless of simple or kerberos authentication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira