You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Tim I <ti...@timisrael.com> on 2016/03/08 03:06:18 UTC
Long running applications in a non-Kerberos cluster?
Hi all,
My previous experience with Slider is in a Kerberized cluster using Slider
0.50.. It required me to restart the apps every 7 days (due to ticket
expiration).
Based on what I've read, I don't think there is anything preventing a
non-Kerberized cluster from running apps indefinitely. Is that correct, or
am I missing something?
I'm currently using Hadoop 2.6.0 if it matters.
Thanks,
Tim
Re: Long running applications in a non-Kerberos cluster?
Posted by Steve Loughran <st...@hortonworks.com>.
> On 10 Mar 2016, at 16:42, Tim I <ti...@timisrael.com> wrote:
>
> Jon,
>
> That's consistent with my experience:
>
> From a kerberos prespective, you can renew a ticket within the renewal
> window (default 24 hours I think)
>
> You can only renew the ticket up to the KDC's setting for 'renew_lifetime'
> (which I believe defaults to 7 days). After that point, you need a new
> ticket via the original credential (keytab, etc..).
>
Oh, it's even more complex than that I'm afraid —there's kerberos tickets and hadoop tokens
tickets: self contained, authenticate a principal with another principal, can be passed on, can be renewed until they expire
hadoop tokens: essentially encrypted indexes into datastructures held by the services; can be refreshed, can be deliberately destroyed (remove whats at the end). Can be passed on. Renewable via IPC or in some cases HTTP(S) —its for different services. . May come with a shorter lifespan than a kerberos ticket, or a longer one.
There's no chatter with the KDC, which has a lot less load on it than otherwise, and there's no need for an HDFS client to auth with every single datanode it may want to talk to.
when you launch an app into a YARN cluster (without a keytab), your kerberos ticket is never bundled. The client gets explicit tokens for the services (HBase, HDFS, ...), adds them to the YARN launch context, then fires it off. They are "magically" picked up at the far end (I'll skip the details, except the magic involves files and the HADOOP_TOKEN_FILE_LOCATION env var), and renewed by the UGI thread until they run out.
When your tokens expire, that's it, unless you are logged in with kerberos via kinit, or can log in via keytab. Which is what it comes down, as tim pointed out. After a few days, no keytab == no fun.
see also
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md
Re: Long running applications in a non-Kerberos cluster?
Posted by Tim I <ti...@timisrael.com>.
Jon,
That's consistent with my experience:
>From a kerberos prespective, you can renew a ticket within the renewal
window (default 24 hours I think)
You can only renew the ticket up to the KDC's setting for 'renew_lifetime'
(which I believe defaults to 7 days). After that point, you need a new
ticket via the original credential (keytab, etc..).
Thanks,
Tim
On Mar 10, 2016 9:45 AM, "Jon Maron" <jm...@hortonworks.com> wrote:
>
> On Mar 9, 2016, at 11:19 PM, Josh Elser <josh.elser@gmail.com<mailto:
> josh.elser@gmail.com>> wrote:
>
> Aww, thanks for the kind words to close it out :). Glad to hear it's
> worked well for you.
>
> Just to share some Kerberos knowledge (and dispel any misinformation), any
> application which stops working after the default ticket lifetime is "doing
> it wrong" (tm). Like Steve pointed out, this is why renewals exist (often a
> dedicated thread running inside the application). Once you have a ticket,
> as long as you ask the KDC for a renewal before that original ticket
> expires, you can get a new one.
>
> Does renewal work past the expiry period? Generally, the default renewal
> period is defined as 24 hours, the default expiry period as days. My
> impression was that you could continually renew every renewal period during
> the expiry period, but ultimately would be required to obtain a new token
> at the 7 day mark:
>
> public static final String
> DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_KEY =
> "dfs.namenode.delegation.token.renew-interval";
> public static final long
> DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT = 24*60*60*1000; // 1
> day
> public static final String DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_KEY
> = "dfs.namenode.delegation.token.max-lifetime";
> public static final long
> DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT = 7*24*60*60*1000; // 7
> days
>
>
> Tim I wrote:
> Great! Thank you all.
>
> Regarding 0.50, it's been a while, but I recall back porting a couple of
> fixes to get my cluster up. IIRC, that was primarily related to kerberos
> fixes for instantiating the cluster.
>
> In 0.50 (+ the patches) accumulo and storm required a restart after 7
> days. Barring that inconvenience, it worked great in a moderately sized
> production environment.
>
> Thanks again everyone.
>
> Tim
> On Mar 9, 2016 8:59 AM, "Jon Maron"<jmaron@hortonworks.com<mailto:
> jmaron@hortonworks.com>> wrote:
>
> On Mar 8, 2016, at 11:36 PM, Tim I<tim@timisrael.com<mailto:
> tim@timisrael.com>> wrote:
>
> Hi Josh,
> Basically anything with a kerberos ticket could no longer could
> communicate
> with anything else after 7 days due to the default config for the
> kerberos
> server :
> renew_lifetime = 7d
>
> The delegation token I believe was the reason for this since it didn't
> have
> access to the original service keytab.
>
> However, what I want to verify is if delegation tokens play any role in
> non-kerberized clusters and if there is anything else that might inhibit
> long running services in that environment.
>
> My suspicion is probably not. I'll continue testing. If anyone knows
> definitively, I'd love to hear about it.
> You are correct - delegation tokens should not play a role in a non-secure
> environment.
>
> What was the specific nature of the issues? i.e. which component ended up
> seeing the expiration issue? there has been work to resolve that issue
> since 0.50.
>
> Thanks!
>
> Tim
> On Mar 8, 2016 11:13 PM, "Josh Elser"<josh.elser@gmail.com<mailto:
> josh.elser@gmail.com>> wrote:
>
> Hi Tim,
>
> I wish I definitively knew the current state of things, but I don't
> anymore. I agree with your assessment though -- there should be no hard
> limit (the current docs state this as requirements too,
> http://slider.incubator.apache.org/docs/security.html).
>
> Was the 7-day expiration you referred to in the Slider app-master? Or
> the
> application Slider was running (e.g. HBase)?
>
> Tim I wrote:
>
> Hi all,
>
> My previous experience with Slider is in a Kerberized cluster using
> Slider
> 0.50.. It required me to restart the apps every 7 days (due to ticket
> expiration).
>
> Based on what I've read, I don't think there is anything preventing a
> non-Kerberized cluster from running apps indefinitely. Is that
> correct,
> or
> am I missing something?
>
> I'm currently using Hadoop 2.6.0 if it matters.
>
> Thanks,
>
> Tim
>
>
>
>
>
>
>
Re: Long running applications in a non-Kerberos cluster?
Posted by Jon Maron <jm...@hortonworks.com>.
On Mar 9, 2016, at 11:19 PM, Josh Elser <jo...@gmail.com>> wrote:
Aww, thanks for the kind words to close it out :). Glad to hear it's worked well for you.
Just to share some Kerberos knowledge (and dispel any misinformation), any application which stops working after the default ticket lifetime is "doing it wrong" (tm). Like Steve pointed out, this is why renewals exist (often a dedicated thread running inside the application). Once you have a ticket, as long as you ask the KDC for a renewal before that original ticket expires, you can get a new one.
Does renewal work past the expiry period? Generally, the default renewal period is defined as 24 hours, the default expiry period as days. My impression was that you could continually renew every renewal period during the expiry period, but ultimately would be required to obtain a new token at the 7 day mark:
public static final String DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_KEY = "dfs.namenode.delegation.token.renew-interval";
public static final long DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT = 24*60*60*1000; // 1 day
public static final String DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_KEY = "dfs.namenode.delegation.token.max-lifetime";
public static final long DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT = 7*24*60*60*1000; // 7 days
Tim I wrote:
Great! Thank you all.
Regarding 0.50, it's been a while, but I recall back porting a couple of
fixes to get my cluster up. IIRC, that was primarily related to kerberos
fixes for instantiating the cluster.
In 0.50 (+ the patches) accumulo and storm required a restart after 7
days. Barring that inconvenience, it worked great in a moderately sized
production environment.
Thanks again everyone.
Tim
On Mar 9, 2016 8:59 AM, "Jon Maron"<jm...@hortonworks.com>> wrote:
On Mar 8, 2016, at 11:36 PM, Tim I<ti...@timisrael.com>> wrote:
Hi Josh,
Basically anything with a kerberos ticket could no longer could
communicate
with anything else after 7 days due to the default config for the
kerberos
server :
renew_lifetime = 7d
The delegation token I believe was the reason for this since it didn't
have
access to the original service keytab.
However, what I want to verify is if delegation tokens play any role in
non-kerberized clusters and if there is anything else that might inhibit
long running services in that environment.
My suspicion is probably not. I'll continue testing. If anyone knows
definitively, I'd love to hear about it.
You are correct - delegation tokens should not play a role in a non-secure
environment.
What was the specific nature of the issues? i.e. which component ended up
seeing the expiration issue? there has been work to resolve that issue
since 0.50.
Thanks!
Tim
On Mar 8, 2016 11:13 PM, "Josh Elser"<jo...@gmail.com>> wrote:
Hi Tim,
I wish I definitively knew the current state of things, but I don't
anymore. I agree with your assessment though -- there should be no hard
limit (the current docs state this as requirements too,
http://slider.incubator.apache.org/docs/security.html).
Was the 7-day expiration you referred to in the Slider app-master? Or
the
application Slider was running (e.g. HBase)?
Tim I wrote:
Hi all,
My previous experience with Slider is in a Kerberized cluster using
Slider
0.50.. It required me to restart the apps every 7 days (due to ticket
expiration).
Based on what I've read, I don't think there is anything preventing a
non-Kerberized cluster from running apps indefinitely. Is that
correct,
or
am I missing something?
I'm currently using Hadoop 2.6.0 if it matters.
Thanks,
Tim
Re: Long running applications in a non-Kerberos cluster?
Posted by Josh Elser <jo...@gmail.com>.
Aww, thanks for the kind words to close it out :). Glad to hear it's
worked well for you.
Just to share some Kerberos knowledge (and dispel any misinformation),
any application which stops working after the default ticket lifetime is
"doing it wrong" (tm). Like Steve pointed out, this is why renewals
exist (often a dedicated thread running inside the application). Once
you have a ticket, as long as you ask the KDC for a renewal before that
original ticket expires, you can get a new one.
Tim I wrote:
> Great! Thank you all.
>
> Regarding 0.50, it's been a while, but I recall back porting a couple of
> fixes to get my cluster up. IIRC, that was primarily related to kerberos
> fixes for instantiating the cluster.
>
> In 0.50 (+ the patches) accumulo and storm required a restart after 7
> days. Barring that inconvenience, it worked great in a moderately sized
> production environment.
>
> Thanks again everyone.
>
> Tim
> On Mar 9, 2016 8:59 AM, "Jon Maron"<jm...@hortonworks.com> wrote:
>
>>> On Mar 8, 2016, at 11:36 PM, Tim I<ti...@timisrael.com> wrote:
>>>
>>> Hi Josh,
>>> Basically anything with a kerberos ticket could no longer could
>> communicate
>>> with anything else after 7 days due to the default config for the
>> kerberos
>>> server :
>>> renew_lifetime = 7d
>>>
>>> The delegation token I believe was the reason for this since it didn't
>> have
>>> access to the original service keytab.
>>>
>>> However, what I want to verify is if delegation tokens play any role in
>>> non-kerberized clusters and if there is anything else that might inhibit
>>> long running services in that environment.
>>>
>>> My suspicion is probably not. I'll continue testing. If anyone knows
>>> definitively, I'd love to hear about it.
>> You are correct - delegation tokens should not play a role in a non-secure
>> environment.
>>
>> What was the specific nature of the issues? i.e. which component ended up
>> seeing the expiration issue? there has been work to resolve that issue
>> since 0.50.
>>
>>> Thanks!
>>>
>>> Tim
>>> On Mar 8, 2016 11:13 PM, "Josh Elser"<jo...@gmail.com> wrote:
>>>
>>>> Hi Tim,
>>>>
>>>> I wish I definitively knew the current state of things, but I don't
>>>> anymore. I agree with your assessment though -- there should be no hard
>>>> limit (the current docs state this as requirements too,
>>>> http://slider.incubator.apache.org/docs/security.html).
>>>>
>>>> Was the 7-day expiration you referred to in the Slider app-master? Or
>> the
>>>> application Slider was running (e.g. HBase)?
>>>>
>>>> Tim I wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> My previous experience with Slider is in a Kerberized cluster using
>> Slider
>>>>> 0.50.. It required me to restart the apps every 7 days (due to ticket
>>>>> expiration).
>>>>>
>>>>> Based on what I've read, I don't think there is anything preventing a
>>>>> non-Kerberized cluster from running apps indefinitely. Is that
>> correct,
>>>>> or
>>>>> am I missing something?
>>>>>
>>>>> I'm currently using Hadoop 2.6.0 if it matters.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Tim
>>>>>
>>>>>
>>
>
Re: Long running applications in a non-Kerberos cluster?
Posted by Tim I <ti...@timisrael.com>.
Great! Thank you all.
Regarding 0.50, it's been a while, but I recall back porting a couple of
fixes to get my cluster up. IIRC, that was primarily related to kerberos
fixes for instantiating the cluster.
In 0.50 (+ the patches) accumulo and storm required a restart after 7
days. Barring that inconvenience, it worked great in a moderately sized
production environment.
Thanks again everyone.
Tim
On Mar 9, 2016 8:59 AM, "Jon Maron" <jm...@hortonworks.com> wrote:
>
> > On Mar 8, 2016, at 11:36 PM, Tim I <ti...@timisrael.com> wrote:
> >
> > Hi Josh,
> > Basically anything with a kerberos ticket could no longer could
> communicate
> > with anything else after 7 days due to the default config for the
> kerberos
> > server :
> > renew_lifetime = 7d
> >
> > The delegation token I believe was the reason for this since it didn't
> have
> > access to the original service keytab.
> >
> > However, what I want to verify is if delegation tokens play any role in
> > non-kerberized clusters and if there is anything else that might inhibit
> > long running services in that environment.
> >
> > My suspicion is probably not. I'll continue testing. If anyone knows
> > definitively, I'd love to hear about it.
>
> You are correct - delegation tokens should not play a role in a non-secure
> environment.
>
> What was the specific nature of the issues? i.e. which component ended up
> seeing the expiration issue? there has been work to resolve that issue
> since 0.50.
>
> >
> > Thanks!
> >
> > Tim
> > On Mar 8, 2016 11:13 PM, "Josh Elser" <jo...@gmail.com> wrote:
> >
> >> Hi Tim,
> >>
> >> I wish I definitively knew the current state of things, but I don't
> >> anymore. I agree with your assessment though -- there should be no hard
> >> limit (the current docs state this as requirements too,
> >> http://slider.incubator.apache.org/docs/security.html).
> >>
> >> Was the 7-day expiration you referred to in the Slider app-master? Or
> the
> >> application Slider was running (e.g. HBase)?
> >>
> >> Tim I wrote:
> >>
> >>> Hi all,
> >>>
> >>> My previous experience with Slider is in a Kerberized cluster using
> Slider
> >>> 0.50.. It required me to restart the apps every 7 days (due to ticket
> >>> expiration).
> >>>
> >>> Based on what I've read, I don't think there is anything preventing a
> >>> non-Kerberized cluster from running apps indefinitely. Is that
> correct,
> >>> or
> >>> am I missing something?
> >>>
> >>> I'm currently using Hadoop 2.6.0 if it matters.
> >>>
> >>> Thanks,
> >>>
> >>> Tim
> >>>
> >>>
>
>
Re: Long running applications in a non-Kerberos cluster?
Posted by Steve Loughran <st...@hortonworks.com>.
> On 9 Mar 2016, at 13:59, Jon Maron <jm...@hortonworks.com> wrote:
>
>>
>> On Mar 8, 2016, at 11:36 PM, Tim I <ti...@timisrael.com> wrote:
>>
>> Hi Josh,
>> Basically anything with a kerberos ticket could no longer could communicate
>> with anything else after 7 days due to the default config for the kerberos
>> server :
>> renew_lifetime = 7d
>>
>> The delegation token I believe was the reason for this since it didn't have
>> access to the original service keytab.
>>
>> However, what I want to verify is if delegation tokens play any role in
>> non-kerberized clusters and if there is anything else that might inhibit
>> long running services in that environment.
>>
>> My suspicion is probably not. I'll continue testing. If anyone knows
>> definitively, I'd love to hear about it.
>
> You are correct - delegation tokens should not play a role in a non-secure environment.
AM/RM tokens are always used; the AM always needs to be updating this every few hours; this ought to be handled automatically
Re: Long running applications in a non-Kerberos cluster?
Posted by Jon Maron <jm...@hortonworks.com>.
> On Mar 8, 2016, at 11:36 PM, Tim I <ti...@timisrael.com> wrote:
>
> Hi Josh,
> Basically anything with a kerberos ticket could no longer could communicate
> with anything else after 7 days due to the default config for the kerberos
> server :
> renew_lifetime = 7d
>
> The delegation token I believe was the reason for this since it didn't have
> access to the original service keytab.
>
> However, what I want to verify is if delegation tokens play any role in
> non-kerberized clusters and if there is anything else that might inhibit
> long running services in that environment.
>
> My suspicion is probably not. I'll continue testing. If anyone knows
> definitively, I'd love to hear about it.
You are correct - delegation tokens should not play a role in a non-secure environment.
What was the specific nature of the issues? i.e. which component ended up seeing the expiration issue? there has been work to resolve that issue since 0.50.
>
> Thanks!
>
> Tim
> On Mar 8, 2016 11:13 PM, "Josh Elser" <jo...@gmail.com> wrote:
>
>> Hi Tim,
>>
>> I wish I definitively knew the current state of things, but I don't
>> anymore. I agree with your assessment though -- there should be no hard
>> limit (the current docs state this as requirements too,
>> http://slider.incubator.apache.org/docs/security.html).
>>
>> Was the 7-day expiration you referred to in the Slider app-master? Or the
>> application Slider was running (e.g. HBase)?
>>
>> Tim I wrote:
>>
>>> Hi all,
>>>
>>> My previous experience with Slider is in a Kerberized cluster using Slider
>>> 0.50.. It required me to restart the apps every 7 days (due to ticket
>>> expiration).
>>>
>>> Based on what I've read, I don't think there is anything preventing a
>>> non-Kerberized cluster from running apps indefinitely. Is that correct,
>>> or
>>> am I missing something?
>>>
>>> I'm currently using Hadoop 2.6.0 if it matters.
>>>
>>> Thanks,
>>>
>>> Tim
>>>
>>>
Re: Long running applications in a non-Kerberos cluster?
Posted by Tim I <ti...@timisrael.com>.
Hi Josh,
Basically anything with a kerberos ticket could no longer could communicate
with anything else after 7 days due to the default config for the kerberos
server :
renew_lifetime = 7d
The delegation token I believe was the reason for this since it didn't have
access to the original service keytab.
However, what I want to verify is if delegation tokens play any role in
non-kerberized clusters and if there is anything else that might inhibit
long running services in that environment.
My suspicion is probably not. I'll continue testing. If anyone knows
definitively, I'd love to hear about it.
Thanks!
Tim
On Mar 8, 2016 11:13 PM, "Josh Elser" <jo...@gmail.com> wrote:
> Hi Tim,
>
> I wish I definitively knew the current state of things, but I don't
> anymore. I agree with your assessment though -- there should be no hard
> limit (the current docs state this as requirements too,
> http://slider.incubator.apache.org/docs/security.html).
>
> Was the 7-day expiration you referred to in the Slider app-master? Or the
> application Slider was running (e.g. HBase)?
>
> Tim I wrote:
>
>> Hi all,
>>
>> My previous experience with Slider is in a Kerberized cluster using Slider
>> 0.50.. It required me to restart the apps every 7 days (due to ticket
>> expiration).
>>
>> Based on what I've read, I don't think there is anything preventing a
>> non-Kerberized cluster from running apps indefinitely. Is that correct,
>> or
>> am I missing something?
>>
>> I'm currently using Hadoop 2.6.0 if it matters.
>>
>> Thanks,
>>
>> Tim
>>
>>
Re: Long running applications in a non-Kerberos cluster?
Posted by Josh Elser <jo...@gmail.com>.
Hi Tim,
I wish I definitively knew the current state of things, but I don't
anymore. I agree with your assessment though -- there should be no hard
limit (the current docs state this as requirements too,
http://slider.incubator.apache.org/docs/security.html).
Was the 7-day expiration you referred to in the Slider app-master? Or
the application Slider was running (e.g. HBase)?
Tim I wrote:
> Hi all,
>
> My previous experience with Slider is in a Kerberized cluster using Slider
> 0.50.. It required me to restart the apps every 7 days (due to ticket
> expiration).
>
> Based on what I've read, I don't think there is anything preventing a
> non-Kerberized cluster from running apps indefinitely. Is that correct, or
> am I missing something?
>
> I'm currently using Hadoop 2.6.0 if it matters.
>
> Thanks,
>
> Tim
>