[jira] [Updated] (YARN-3439) RM fails to renew token when Oozie launcher leaves before sub-job finishes

["Jason Lowe (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Lowe updated YARN-3439:
-----------------------------
    Attachment: YARN-3439.001.patch

Daryn is out so posting a prototype patch he developed to get some early feedback.  Note that this patch can't go in as-is, as it's a work-in-progress that hacks out the automatic HDFS delegation token logic that was added as part of YARN-2704.

Essentially the idea is to reference count the tokens and only attempt to cancel them when the token is no longer referenced.  Since the launcher job won't complete until it has successfully submitted the sub-job(s), the token will remain referenced throughout the lifespan of the workflow even if the launcher job exits early.

> RM fails to renew token when Oozie launcher leaves before sub-job finishes
> --------------------------------------------------------------------------
>
>                 Key: YARN-3439
>                 URL: https://issues.apache.org/jira/browse/YARN-3439
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.7.0
>            Reporter: Jason Lowe
>            Assignee: Daryn Sharp
>            Priority: Blocker
>         Attachments: YARN-3439.001.patch
>
>
> When the Oozie launcher runs a standard MapReduce job (not Pig) it doesn't linger waiting for the sub-job to finish.  At that point the RM stops renewing delegation tokens for the launcher job which wreaks havoc on the sub-job if the sub-job runs long enough for the tokens to expire.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)