You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by "Daminato,Josh" <Jo...@cerner.com.INVALID> on 2018/06/13 15:24:21 UTC
Delegation Token Expiring
We recently ran into an issue where our launcher task attempted to kick off map reduce jobs after the delegation token provided by Oozie had expired.
We found that we could increase 'yarn.resourcemanager.delegation.token.renew-interval', but we also started thinking that maybe it made sense for Oozie itself to renew these tokens.
Oozie is already monitoring the Java actions that it kicks off, so we thought why not have it also keep the delegation tokens that it provided to that action alive while the action is still running.
We are currently running without Kerberos enabled, and on 4.1.0 version of Oozie.
I fiddled around with renewing the token programmatically in the launcher task, and was able to get it working by pretending to be the fake 'oozie mr token' user that Oozie sets as the renewer in an insecure cluster. But switching to that user to renew a delegation token is a hack.
I also experimented briefly on a cluster with Kerberos enabled, and I found that Oozie set 'yarn' as the renewer of the 'RM_DELEGATION_TOKEN'. Not sure why this is. Will the resource manager renew this token?
Curious on anyones thoughts about Oozie automagically renewing the delegation tokens that it passes to Java actions while the actions are still running.
Thanks,
Josh
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
Re: Delegation Token Expiring
Posted by Peter Cseh <ge...@cloudera.com.INVALID>.
Hey!
The credentials.skip affect's Oozie's credential implementations only,
which is in 4.1.0 is Hcat, Hive and Hive2. Other tokens are acquired by the
MapReduce client (like AMRM token, HDFS token, JHS token). In Oozie 5.0 and
above all tokens are affected by the credentials.skip property.
If you're not using kerberos anyways, I suggest just to increase the
lifetime of the ticket via the config property you've found to something
that will last long enough.
GP
On Mon, Jun 18, 2018 at 6:56 PM Daminato,Josh
<Jo...@cerner.com.invalid> wrote:
> Thanks for your quick response, and sorry for my slow one!
>
>
> Yeah that blog post is super helpful.
>
>
> I should have looked at the master Oozie code more. Looking at it now, I
> can see what you're talking about with the token not being set when
> Kerberos is not enabled.
>
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1019
>
>
> While looking around in there, I also saw the 'oozie.credentials.skip'
> property. I hadn't seen that in my initial investigation.
>
> The code (even for our older version) seems to suggest that by setting it
> Oozie wouldn't try and set a delegation token.
>
> I tried it out, and I can see "Skipping credentials" in the Oozie service
> logs, but I still see the delegation token in my launcher.
>
>
> Maybe I'm misunderstanding that configuration, and it's only for ignoring
> a credentials section in your workflow?
>
>
> Thanks again for your response!
>
>
>
> ________________________________
> From: Peter Cseh <ge...@cloudera.com.INVALID>
> Sent: Thursday, June 14, 2018 2:20:12 AM
> To: user@oozie.apache.org
> Subject: Re: Delegation Token Expiring
>
> Hey!
>
> Yes, there were some issues about expiring tokens in a non-kerberized
> environment. They should be fixed in Oozie 5.0 where we don't get tokens if
> they are not required.
> By setting the renewer to "yarn" we allow the ResourceManager to renew our
> delegation tokens.
> This blogpost explains a lot about this issue:
>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.cloudera.com%2Fblog%2F2017%2F12%2Fhadoop-delegation-tokens-explained%2F&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=X9f3KxOgsag%2FTqSGHnZjIMVdBPOZIFGt4uFnYkO6I3c%3D&reserved=0
>
> Hope it helps,
> gp
>
> On Wed, Jun 13, 2018 at 5:24 PM Daminato,Josh
> <Jo...@cerner.com.invalid> wrote:
>
> > We recently ran into an issue where our launcher task attempted to kick
> > off map reduce jobs after the delegation token provided by Oozie had
> > expired.
> >
> > We found that we could increase
> > 'yarn.resourcemanager.delegation.token.renew-interval', but we also
> started
> > thinking that maybe it made sense for Oozie itself to renew these tokens.
> >
> > Oozie is already monitoring the Java actions that it kicks off, so we
> > thought why not have it also keep the delegation tokens that it provided
> to
> > that action alive while the action is still running.
> >
> > We are currently running without Kerberos enabled, and on 4.1.0 version
> of
> > Oozie.
> >
> > I fiddled around with renewing the token programmatically in the launcher
> > task, and was able to get it working by pretending to be the fake 'oozie
> mr
> > token' user that Oozie sets as the renewer in an insecure cluster. But
> > switching to that user to renew a delegation token is a hack.
> >
> > I also experimented briefly on a cluster with Kerberos enabled, and I
> > found that Oozie set 'yarn' as the renewer of the 'RM_DELEGATION_TOKEN'.
> > Not sure why this is. Will the resource manager renew this token?
> >
> >
> > Curious on anyones thoughts about Oozie automagically renewing the
> > delegation tokens that it passes to Java actions while the actions are
> > still running.
> >
> >
> > Thanks,
> > Josh
> >
> >
> >
> >
> > CONFIDENTIALITY NOTICE This message and any included attachments are from
> > Cerner Corporation and are intended only for the addressee. The
> information
> > contained in this message is confidential and may constitute inside or
> > non-public information under international, federal, or state securities
> > laws. Unauthorized forwarding, printing, copying, distribution, or use of
> > such information is strictly prohibited and may be unlawful. If you are
> not
> > the addressee, please promptly delete this message and notify the sender
> of
> > the delivery error by e-mail or you may call Cerner's corporate offices
> in
> > Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
> >
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com <
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cloudera.com&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=CBJ1suysvgX78dXuPO5FuxfHaIHP%2BXNpAI7sQh6m630%3D&reserved=0
> >
>
> [image: Cloudera] <
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cloudera.com%2F&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=EKFgAA4LvqbrxF%2FjPxrrThThwYwm8EikozJoDXhLZr4%3D&reserved=0
> >
>
> [image: Cloudera on Twitter] <
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fcloudera&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=TsU1o8x5qF8EYqcrhtkn%2BPAWqj5KJZsyQtvICIEFCB4%3D&reserved=0>
> [image:
> Cloudera on Facebook] <
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fcloudera&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=EcJgkhOigrv8UGNDAFuEy2xEg7rmbqe30%2BH5%2BrMIwpE%3D&reserved=0>
> [image: Cloudera
> on LinkedIn] <
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fcloudera&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=lIaujH%2BpeqnpqqMtedIKHGtQFqCow3LPl0CkSzRjGsY%3D&reserved=0
> >
> ------------------------------
>
>
> CONFIDENTIALITY NOTICE This message and any included attachments are from
> Cerner Corporation and are intended only for the addressee. The information
> contained in this message is confidential and may constitute inside or
> non-public information under international, federal, or state securities
> laws. Unauthorized forwarding, printing, copying, distribution, or use of
> such information is strictly prohibited and may be unlawful. If you are not
> the addressee, please promptly delete this message and notify the sender of
> the delivery error by e-mail or you may call Cerner's corporate offices in
> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>
--
*Peter Cseh *| Software Engineer
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
Re: Delegation Token Expiring
Posted by "Daminato,Josh" <Jo...@cerner.com.INVALID>.
Thanks for your quick response, and sorry for my slow one!
Yeah that blog post is super helpful.
I should have looked at the master Oozie code more. Looking at it now, I can see what you're talking about with the token not being set when Kerberos is not enabled.
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1019
While looking around in there, I also saw the 'oozie.credentials.skip' property. I hadn't seen that in my initial investigation.
The code (even for our older version) seems to suggest that by setting it Oozie wouldn't try and set a delegation token.
I tried it out, and I can see "Skipping credentials" in the Oozie service logs, but I still see the delegation token in my launcher.
Maybe I'm misunderstanding that configuration, and it's only for ignoring a credentials section in your workflow?
Thanks again for your response!
________________________________
From: Peter Cseh <ge...@cloudera.com.INVALID>
Sent: Thursday, June 14, 2018 2:20:12 AM
To: user@oozie.apache.org
Subject: Re: Delegation Token Expiring
Hey!
Yes, there were some issues about expiring tokens in a non-kerberized
environment. They should be fixed in Oozie 5.0 where we don't get tokens if
they are not required.
By setting the renewer to "yarn" we allow the ResourceManager to renew our
delegation tokens.
This blogpost explains a lot about this issue:
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.cloudera.com%2Fblog%2F2017%2F12%2Fhadoop-delegation-tokens-explained%2F&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=X9f3KxOgsag%2FTqSGHnZjIMVdBPOZIFGt4uFnYkO6I3c%3D&reserved=0
Hope it helps,
gp
On Wed, Jun 13, 2018 at 5:24 PM Daminato,Josh
<Jo...@cerner.com.invalid> wrote:
> We recently ran into an issue where our launcher task attempted to kick
> off map reduce jobs after the delegation token provided by Oozie had
> expired.
>
> We found that we could increase
> 'yarn.resourcemanager.delegation.token.renew-interval', but we also started
> thinking that maybe it made sense for Oozie itself to renew these tokens.
>
> Oozie is already monitoring the Java actions that it kicks off, so we
> thought why not have it also keep the delegation tokens that it provided to
> that action alive while the action is still running.
>
> We are currently running without Kerberos enabled, and on 4.1.0 version of
> Oozie.
>
> I fiddled around with renewing the token programmatically in the launcher
> task, and was able to get it working by pretending to be the fake 'oozie mr
> token' user that Oozie sets as the renewer in an insecure cluster. But
> switching to that user to renew a delegation token is a hack.
>
> I also experimented briefly on a cluster with Kerberos enabled, and I
> found that Oozie set 'yarn' as the renewer of the 'RM_DELEGATION_TOKEN'.
> Not sure why this is. Will the resource manager renew this token?
>
>
> Curious on anyones thoughts about Oozie automagically renewing the
> delegation tokens that it passes to Java actions while the actions are
> still running.
>
>
> Thanks,
> Josh
>
>
>
>
> CONFIDENTIALITY NOTICE This message and any included attachments are from
> Cerner Corporation and are intended only for the addressee. The information
> contained in this message is confidential and may constitute inside or
> non-public information under international, federal, or state securities
> laws. Unauthorized forwarding, printing, copying, distribution, or use of
> such information is strictly prohibited and may be unlawful. If you are not
> the addressee, please promptly delete this message and notify the sender of
> the delivery error by e-mail or you may call Cerner's corporate offices in
> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>
--
*Peter Cseh *| Software Engineer
cloudera.com <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cloudera.com&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=CBJ1suysvgX78dXuPO5FuxfHaIHP%2BXNpAI7sQh6m630%3D&reserved=0>
[image: Cloudera] <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cloudera.com%2F&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=EKFgAA4LvqbrxF%2FjPxrrThThwYwm8EikozJoDXhLZr4%3D&reserved=0>
[image: Cloudera on Twitter] <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fcloudera&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=TsU1o8x5qF8EYqcrhtkn%2BPAWqj5KJZsyQtvICIEFCB4%3D&reserved=0> [image:
Cloudera on Facebook] <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fcloudera&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=EcJgkhOigrv8UGNDAFuEy2xEg7rmbqe30%2BH5%2BrMIwpE%3D&reserved=0> [image: Cloudera
on LinkedIn] <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fcloudera&data=02%7C01%7CJosh.Daminato%40cerner.com%7Ccce57e869098436b36b608d5d1c761cd%7Cfbc493a80d244454a815f4ca58e8c09d%7C0%7C0%7C636645576628739284&sdata=lIaujH%2BpeqnpqqMtedIKHGtQFqCow3LPl0CkSzRjGsY%3D&reserved=0>
------------------------------
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
Re: Delegation Token Expiring
Posted by Peter Cseh <ge...@cloudera.com.INVALID>.
Hey!
Yes, there were some issues about expiring tokens in a non-kerberized
environment. They should be fixed in Oozie 5.0 where we don't get tokens if
they are not required.
By setting the renewer to "yarn" we allow the ResourceManager to renew our
delegation tokens.
This blogpost explains a lot about this issue:
https://blog.cloudera.com/blog/2017/12/hadoop-delegation-tokens-explained/
Hope it helps,
gp
On Wed, Jun 13, 2018 at 5:24 PM Daminato,Josh
<Jo...@cerner.com.invalid> wrote:
> We recently ran into an issue where our launcher task attempted to kick
> off map reduce jobs after the delegation token provided by Oozie had
> expired.
>
> We found that we could increase
> 'yarn.resourcemanager.delegation.token.renew-interval', but we also started
> thinking that maybe it made sense for Oozie itself to renew these tokens.
>
> Oozie is already monitoring the Java actions that it kicks off, so we
> thought why not have it also keep the delegation tokens that it provided to
> that action alive while the action is still running.
>
> We are currently running without Kerberos enabled, and on 4.1.0 version of
> Oozie.
>
> I fiddled around with renewing the token programmatically in the launcher
> task, and was able to get it working by pretending to be the fake 'oozie mr
> token' user that Oozie sets as the renewer in an insecure cluster. But
> switching to that user to renew a delegation token is a hack.
>
> I also experimented briefly on a cluster with Kerberos enabled, and I
> found that Oozie set 'yarn' as the renewer of the 'RM_DELEGATION_TOKEN'.
> Not sure why this is. Will the resource manager renew this token?
>
>
> Curious on anyones thoughts about Oozie automagically renewing the
> delegation tokens that it passes to Java actions while the actions are
> still running.
>
>
> Thanks,
> Josh
>
>
>
>
> CONFIDENTIALITY NOTICE This message and any included attachments are from
> Cerner Corporation and are intended only for the addressee. The information
> contained in this message is confidential and may constitute inside or
> non-public information under international, federal, or state securities
> laws. Unauthorized forwarding, printing, copying, distribution, or use of
> such information is strictly prohibited and may be unlawful. If you are not
> the addressee, please promptly delete this message and notify the sender of
> the delivery error by e-mail or you may call Cerner's corporate offices in
> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>
--
*Peter Cseh *| Software Engineer
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------