You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Jon Maron <jm...@hortonworks.com> on 2014/08/17 00:18:37 UTC
Slider Token Renewal for long running apps
Hi,
I am going to try to describe an issue I’ve identified during the implementation of a Slider mechanism for renewing tokens and replacing expired tokens. Here goes…
Cluster setup:
- Secure Hadoop Cluster
- User ‘jon’ defined as both a host user on the cluster nodes and a kerberos principal (jon@EXAMPLE.COM)
- ‘jon’ is the user that is invoking the slider CLI and launching the applications. Once launched, the UGI/Subject in the AM is ‘jon’
- It’s also important to note that, in the runtime, the FileSystem reference is associated with user ‘jon’ (I believe user name is part of the key for the file system reference cache)
Token renewal:
- Token renewal for HDFS is going to require the invocation of a number FileSystem APIs:
- FileSystem.getDelegationToken(renewer)
- FileSystem.renew()
- FileSystem.addDelegationTokens()
- It turns out that, to be invoked, these APIs require a Kerberos authenticated subject rather than a token authenticated subject (a subject whose auth method is TOKEN)
- My initial thought was to leverage the NN keytab (kerberos principal nn@_HOST@EXAMPLE.COM mapped to ‘hdfs’) to create a subject/UGI to perform the API invocations (UGI..loginUserFromKeytabAndReturnUGI())
- I also couldn’t use the FileSystem reference maintained in the AM’s CoreFileSystem because it was associated with the subject ‘jon’ that was authenticated by token. I attempted to work around this issue by explicitly creating and using a FileSystem reference associated with user ‘hdfs’ to perform the token operations. However, that triggered another issue…
- Once a FileSystem existed for user ‘hdfs’ in the file system cache, invocations of FileSystem.get(Configuration) (which occur in the runtime) could return the ‘hdfs’ FileSystem reference, which can trigger these sort of mixed-user paths:
drwxr-xr-x - jon users 0 2014-08-16 20:29 /user/jon/.slider/cluster/cl1/tmp/application_1408124780611_0016
drwxr-xr-x - hdfs users 0 2014-08-16 20:29 /user/jon/.slider/cluster/cl1/tmp/appmaster
These sort of mixed user paths yield bizarre “I can’t update this cluster” issues etc
My suggested solution:
I believe the best course for avoiding the mixed-user issues and the “renewer doesn’t match the file system user” issues is to actually create a Slider keytab and populate it with the principals that will be creating applications using Slider, e.g., in the case above the slider.keytab would have the jon@EXAMPLE.com principal and the default principal to auth mapping will map that user to ‘jon’ on the host machines. I think this would allow for clearly identifying the identity responsible for token renewal and would align the launching identity with the FileSystem identity. I also believe this aligns Slider with the secure practices of other Hadoop elements.
— Jon
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Slider Token Renewal for long running apps
Posted by Jon Maron <jm...@hortonworks.com>.
Going Mobile
> On Aug 17, 2014, at 5:35 PM, Sumit Mohanty <sm...@hortonworks.com> wrote:
>
> So how does the solution look for two users? Is it a single keytab with two
> principals.
That would be my approach.
> What sort of visibility each user will have of resources of the
> other user?
Well - each AM instance would only be associated with a single user at a time. If a situation did somehow arise where more than one user was logged in it would still work since the tokens are credentials associated with a given subject. We would, however, require a more robust renewal mechanism that could handle multiple token renewal periods.
>
>
>> On Sun, Aug 17, 2014 at 1:01 PM, Jon Maron <jm...@hortonworks.com> wrote:
>>
>>
>> On Aug 17, 2014, at 10:35 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>>> On 16 August 2014 23:18, Jon Maron <jm...@hortonworks.com> wrote:
>>>>
>>>> Once a FileSystem existed for user ‘hdfs’ in the file system cache,
>>>> invocations of FileSystem.get(Configuration) (
>>>
>>>
>>> you can turn the cache off for a filesystem, this gives you a new
>>> instance.. look in FileSystem.get():
>>>
>>> String disableCacheName = String.format("fs.%s.impl.disable.cache",
>>> scheme);
>>> if (conf.getBoolean(disableCacheName, false)) {
>>> return createFileSystem(uri, conf);
>>> }
>>>
>>> i.e add to the conf
>>>
>>> fs.hdfs.impl.disable.cache=true
>>>
>>> and you get a new one
>>
>> I suppose that would address the mixed-user issue - the hdfs principal
>> would only be leveraged for the token related invocations and the instance
>> would not be accessible to the app identity (‘jon’). But is that a
>> configuration setting we could require users to leverage in a secure
>> deployment?
>>
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>> immediately
>>> and delete it from your system. Thank You.
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Slider Token Renewal for long running apps
Posted by Sumit Mohanty <sm...@hortonworks.com>.
So how does the solution look for two users? Is it a single keytab with two
principals. What sort of visibility each user will have of resources of the
other user?
On Sun, Aug 17, 2014 at 1:01 PM, Jon Maron <jm...@hortonworks.com> wrote:
>
> On Aug 17, 2014, at 10:35 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > On 16 August 2014 23:18, Jon Maron <jm...@hortonworks.com> wrote:
> >
> >> Once a FileSystem existed for user ‘hdfs’ in the file system cache,
> >> invocations of FileSystem.get(Configuration) (
> >
> >
> > you can turn the cache off for a filesystem, this gives you a new
> > instance.. look in FileSystem.get():
> >
> > String disableCacheName = String.format("fs.%s.impl.disable.cache",
> > scheme);
> > if (conf.getBoolean(disableCacheName, false)) {
> > return createFileSystem(uri, conf);
> > }
> >
> > i.e add to the conf
> >
> > fs.hdfs.impl.disable.cache=true
> >
> > and you get a new one
>
> I suppose that would address the mixed-user issue - the hdfs principal
> would only be leveraged for the token related invocations and the instance
> would not be accessible to the app identity (‘jon’). But is that a
> configuration setting we could require users to leverage in a secure
> deployment?
>
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Slider Token Renewal for long running apps
Posted by Jon Maron <jm...@hortonworks.com>.
On Aug 18, 2014, at 5:02 AM, Steve Loughran <st...@hortonworks.com> wrote:
> we can just flip the bit on the config you want to use to create the hdfs
> account ... take the normal config, patch it, then ask for the FS. I've
> used this in the past; it works happily.
I see. I’m going to give that a shot. I’ll reuse that config for the token renewal/expiry interactions and see if the scheme woks.
>
> It's main role is get by code that calls FileSystem.close() while other
> clients are using it.
>
>
> On 17 August 2014 21:01, Jon Maron <jm...@hortonworks.com> wrote:
>
>>
>> On Aug 17, 2014, at 10:35 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>> On 16 August 2014 23:18, Jon Maron <jm...@hortonworks.com> wrote:
>>>
>>>> Once a FileSystem existed for user ‘hdfs’ in the file system cache,
>>>> invocations of FileSystem.get(Configuration) (
>>>
>>>
>>> you can turn the cache off for a filesystem, this gives you a new
>>> instance.. look in FileSystem.get():
>>>
>>> String disableCacheName = String.format("fs.%s.impl.disable.cache",
>>> scheme);
>>> if (conf.getBoolean(disableCacheName, false)) {
>>> return createFileSystem(uri, conf);
>>> }
>>>
>>> i.e add to the conf
>>>
>>> fs.hdfs.impl.disable.cache=true
>>>
>>> and you get a new one
>>
>> I suppose that would address the mixed-user issue - the hdfs principal
>> would only be leveraged for the token related invocations and the instance
>> would not be accessible to the app identity (‘jon’). But is that a
>> configuration setting we could require users to leverage in a secure
>> deployment?
>>
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>> immediately
>>> and delete it from your system. Thank You.
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Slider Token Renewal for long running apps
Posted by Steve Loughran <st...@hortonworks.com>.
we can just flip the bit on the config you want to use to create the hdfs
account ... take the normal config, patch it, then ask for the FS. I've
used this in the past; it works happily.
It's main role is get by code that calls FileSystem.close() while other
clients are using it.
On 17 August 2014 21:01, Jon Maron <jm...@hortonworks.com> wrote:
>
> On Aug 17, 2014, at 10:35 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > On 16 August 2014 23:18, Jon Maron <jm...@hortonworks.com> wrote:
> >
> >> Once a FileSystem existed for user ‘hdfs’ in the file system cache,
> >> invocations of FileSystem.get(Configuration) (
> >
> >
> > you can turn the cache off for a filesystem, this gives you a new
> > instance.. look in FileSystem.get():
> >
> > String disableCacheName = String.format("fs.%s.impl.disable.cache",
> > scheme);
> > if (conf.getBoolean(disableCacheName, false)) {
> > return createFileSystem(uri, conf);
> > }
> >
> > i.e add to the conf
> >
> > fs.hdfs.impl.disable.cache=true
> >
> > and you get a new one
>
> I suppose that would address the mixed-user issue - the hdfs principal
> would only be leveraged for the token related invocations and the instance
> would not be accessible to the app identity (‘jon’). But is that a
> configuration setting we could require users to leverage in a secure
> deployment?
>
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Slider Token Renewal for long running apps
Posted by Jon Maron <jm...@hortonworks.com>.
On Aug 17, 2014, at 10:35 AM, Steve Loughran <st...@hortonworks.com> wrote:
> On 16 August 2014 23:18, Jon Maron <jm...@hortonworks.com> wrote:
>
>> Once a FileSystem existed for user ‘hdfs’ in the file system cache,
>> invocations of FileSystem.get(Configuration) (
>
>
> you can turn the cache off for a filesystem, this gives you a new
> instance.. look in FileSystem.get():
>
> String disableCacheName = String.format("fs.%s.impl.disable.cache",
> scheme);
> if (conf.getBoolean(disableCacheName, false)) {
> return createFileSystem(uri, conf);
> }
>
> i.e add to the conf
>
> fs.hdfs.impl.disable.cache=true
>
> and you get a new one
I suppose that would address the mixed-user issue - the hdfs principal would only be leveraged for the token related invocations and the instance would not be accessible to the app identity (‘jon’). But is that a configuration setting we could require users to leverage in a secure deployment?
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Slider Token Renewal for long running apps
Posted by Steve Loughran <st...@hortonworks.com>.
On 16 August 2014 23:18, Jon Maron <jm...@hortonworks.com> wrote:
> Once a FileSystem existed for user ‘hdfs’ in the file system cache,
> invocations of FileSystem.get(Configuration) (
you can turn the cache off for a filesystem, this gives you a new
instance.. look in FileSystem.get():
String disableCacheName = String.format("fs.%s.impl.disable.cache",
scheme);
if (conf.getBoolean(disableCacheName, false)) {
return createFileSystem(uri, conf);
}
i.e add to the conf
fs.hdfs.impl.disable.cache=true
and you get a new one
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.