You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by "Menon, Rohit" <rm...@ea.com> on 2017/06/05 19:28:31 UTC

Oozie 4.3.0 Launcher Uber Mode + Preemption

Hello,

I had a query regarding https://issues.apache.org/jira/browse/OOZIE-2123.

We want to enable uber mode for oozie launchers as a prerequisite for preemption. However we do use mapreduce.user.classpath.first configuration in our environment. What is the reasoning/concern behind disabling uber mode when the above configuration is set?

Any other suggestions for using preemption with oozie?

We are using oozie 4.3.0 + Hadoop 2.7.3

Thanks,
Rohit

Re: Oozie 4.3.0 Launcher Uber Mode + Preemption

Posted by "Menon, Rohit" <rm...@ea.com>.
Hey Andras,

Thank you for your response.

I understand that mapreduce.job.classloader or mapreduce.user.classpath.first takes precendence over mapreduce.job.ubertask.enable. However why is this precedence enforced? What are the concerns? I could not find any details in the ticket. 

This is a blocker for us, since the configurations (mapreduce.job.classloader or mapreduce.user.classpath.first) are cluster wide and cannot be overridden at oozie server level. We could ideally override these properties at each workflow action level, however that would mean making changes to every running coordinator in the cluster.

Thanks for pointing to the Oozie YARN ticket. However there seems to be many open issues as per https://issues.apache.org/jira/browse/OOZIE-2889. Also 5.0.0 not being GA, we cannot use it.

Regards,
Rohit

On 6/6/17, 5:12 AM, "Andras Piros" <an...@cloudera.com> wrote:

    Hi Rohit,
    
    OOZIE-2123 <https://issues.apache.org/jira/browse/OOZIE-2123> is that when
    those parameters mapreduce.job.classloader or mapreduce.user.classpath.first
    are set by the user, they take precedence over oozie.action.launcher.
    mapreduce.job.ubertask.enable one - it will be silently set to false. The
    reasoning behind is that those parameters should have no effects when the
    Oozie launcher map-only MapTask runs in the same JVM of the YARN
    ApplicationMaster.
    
    Actually Oozie On YARN <https://issues.apache.org/jira/browse/OOZIE-1770> is
    already merged to master, meaning this uber job enable switch won't have an
    effect for the upcoming Oozie 5.0.0.
    
    As for how to influence YARN AM preemption, please see the
    CapacityScheduler documentation
    <https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html>.
    on how to use yarn.scheduler.capacity.maximum-am-resource-percent.
    
    Regards,
    
    Andras
    
    On Mon, Jun 5, 2017 at 9:28 PM, Menon, Rohit <rm...@ea.com> wrote:
    
    > Hello,
    >
    > I had a query regarding https://issues.apache.org/jira/browse/OOZIE-2123.
    >
    > We want to enable uber mode for oozie launchers as a prerequisite for
    > preemption. However we do use mapreduce.user.classpath.first configuration
    > in our environment. What is the reasoning/concern behind disabling uber
    > mode when the above configuration is set?
    >
    > Any other suggestions for using preemption with oozie?
    >
    > We are using oozie 4.3.0 + Hadoop 2.7.3
    >
    > Thanks,
    > Rohit
    >

Re: Oozie 4.3.0 Launcher Uber Mode + Preemption

Posted by Andras Piros <an...@cloudera.com>.
Hi Rohit,

OOZIE-2123 <https://issues.apache.org/jira/browse/OOZIE-2123> is that when
those parameters mapreduce.job.classloader or mapreduce.user.classpath.first
are set by the user, they take precedence over oozie.action.launcher.
mapreduce.job.ubertask.enable one - it will be silently set to false. The
reasoning behind is that those parameters should have no effects when the
Oozie launcher map-only MapTask runs in the same JVM of the YARN
ApplicationMaster.

Actually Oozie On YARN <https://issues.apache.org/jira/browse/OOZIE-1770> is
already merged to master, meaning this uber job enable switch won't have an
effect for the upcoming Oozie 5.0.0.

As for how to influence YARN AM preemption, please see the
CapacityScheduler documentation
<https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html>.
on how to use yarn.scheduler.capacity.maximum-am-resource-percent.

Regards,

Andras

On Mon, Jun 5, 2017 at 9:28 PM, Menon, Rohit <rm...@ea.com> wrote:

> Hello,
>
> I had a query regarding https://issues.apache.org/jira/browse/OOZIE-2123.
>
> We want to enable uber mode for oozie launchers as a prerequisite for
> preemption. However we do use mapreduce.user.classpath.first configuration
> in our environment. What is the reasoning/concern behind disabling uber
> mode when the above configuration is set?
>
> Any other suggestions for using preemption with oozie?
>
> We are using oozie 4.3.0 + Hadoop 2.7.3
>
> Thanks,
> Rohit
>