You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Bijoy Deb <bi...@gmail.com> on 2016/09/10 17:37:59 UTC
Unable to configure custom mapreduce.job.queuename in Oozie shell action
Hi,
I have configured 2 new queues 'launcher' and 'mapreduce' apart from
'default' in Capacity scheduler in yarn-site.
Now, I am trying to submit a Pig job via *Oozie Shell action* into those
queues queues such that the oozie launcher job goes into 'launcher' and pig
job into 'mapreduce' queue.On submitting the oozie workflow I can see that
my launcher vjob successfully runs in 'launcher' queue,but the
Pig/mapreduce job is still running in default queue.
Below is the snippet that I added to my workflow.xml for above:
*<configuration>*
*<property>*
*<name>oozie.launcher.mapreduce.job.queuename</name>*
*<value>launcher</value>*
*</property>*
*<property>*
*<name>mapreduce.job.queuename</name>*
*<value>mapreduce</value>*
*</property>*
*</configuration>*
I also tried using 'mapred.job.queue.name' instead of
'mapreduce.job.queuename' above and also by passing it as
-Dmapreduce.job.queuename=mapreduce while submitting the oozie job via
Oozie cli.But in every case the Pig/mapreduce job goes into 'default' queue.
Am I doing anything wrong here or is it that Oozie Shell action doesn't
support 'mapreduce.job.quenename specification?
I am using MR2, Hadoop version 2.7.1, oozie version 4.2.0.
Any help would be really appreciated.
Thanks
Re: Unable to configure custom mapreduce.job.queuename in Oozie shell action
Posted by Bijoy Deb <bi...@gmail.com>.
Hi,
We are having multiple pig scripts and hive scripts that we are going to
call from a generic oozie workflow.So,specifying the property in oozie
action would be more preferable to me.
On Tue, Sep 13, 2016 at 2:10 PM, Abhishek Bafna <ba...@gmail.com>
wrote:
> Hi,
>
> Then why not use the same way of specifying the queue name in the pig
> script and launch it.
> set mapreduce.job.queuename=queuename
>
> ~Abhishek
> > On Sep 13, 2016, at 1:48 PM, Bijoy Deb <bi...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > When I submit a job using Pig command,I am able to submit it to the
> custom
> > queue using 'set mapreduce.job.queuename <name_of_queue>' command inside
> > Pig script.It seems the problem is when I specify the queuename in
> > <configuration> tag via Oozie shell action.
> >
> > We cannot use the Pig script directly and use shell action instead as we
> > have some custom operations such as logging,auditing,etc. that we want to
> > perform via the shell script during the course of execution of the Pig
> job
> > from within it.
> >
> > Thanks
> >
> > On Mon, Sep 12, 2016 at 1:44 AM, Peter Cseh <ge...@cloudera.com>
> wrote:
> >
> >> Hi,
> >>
> >> Can you submit jobs into the specified queues outside Oozie?
> >> Is there any reason Pig Action
> >> <https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#
> >> a3.2.3_Pig_Action>
> >> can't
> >> be used instead of Shell Action to submit Pig jobs?
> >>
> >> Thanks,
> >> Peter
> >>
> >>
> >>
> >> On Sat, Sep 10, 2016 at 7:37 PM, Bijoy Deb <bi...@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>> I have configured 2 new queues 'launcher' and 'mapreduce' apart from
> >>> 'default' in Capacity scheduler in yarn-site.
> >>> Now, I am trying to submit a Pig job via *Oozie Shell action* into
> those
> >>> queues queues such that the oozie launcher job goes into 'launcher' and
> >> pig
> >>> job into 'mapreduce' queue.On submitting the oozie workflow I can see
> >> that
> >>> my launcher vjob successfully runs in 'launcher' queue,but the
> >>> Pig/mapreduce job is still running in default queue.
> >>>
> >>> Below is the snippet that I added to my workflow.xml for above:
> >>>
> >>> *<configuration>*
> >>>
> >>> *<property>*
> >>>
> >>> *<name>oozie.launcher.mapreduce.job.queuename</name>*
> >>>
> >>> *<value>launcher</value>*
> >>>
> >>> *</property>*
> >>>
> >>> *<property>*
> >>>
> >>> *<name>mapreduce.job.queuename</name>*
> >>>
> >>> *<value>mapreduce</value>*
> >>> *</property>*
> >>> *</configuration>*
> >>>
> >>> I also tried using 'mapred.job.queue.name' instead of
> >>> 'mapreduce.job.queuename' above and also by passing it as
> >>> -Dmapreduce.job.queuename=mapreduce while submitting the oozie job via
> >>> Oozie cli.But in every case the Pig/mapreduce job goes into 'default'
> >>> queue.
> >>>
> >>> Am I doing anything wrong here or is it that Oozie Shell action doesn't
> >>> support 'mapreduce.job.quenename specification?
> >>>
> >>> I am using MR2, Hadoop version 2.7.1, oozie version 4.2.0.
> >>>
> >>> Any help would be really appreciated.
> >>>
> >>> Thanks
> >>>
> >>
> >>
> >>
> >> --
> >> Peter Cseh
> >> Software Engineer
> >> <http://www.cloudera.com>
> >>
>
>
Re: Unable to configure custom mapreduce.job.queuename in Oozie shell action
Posted by Abhishek Bafna <ba...@gmail.com>.
Hi,
Then why not use the same way of specifying the queue name in the pig script and launch it.
set mapreduce.job.queuename=queuename
~Abhishek
> On Sep 13, 2016, at 1:48 PM, Bijoy Deb <bi...@gmail.com> wrote:
>
> Hi,
>
> When I submit a job using Pig command,I am able to submit it to the custom
> queue using 'set mapreduce.job.queuename <name_of_queue>' command inside
> Pig script.It seems the problem is when I specify the queuename in
> <configuration> tag via Oozie shell action.
>
> We cannot use the Pig script directly and use shell action instead as we
> have some custom operations such as logging,auditing,etc. that we want to
> perform via the shell script during the course of execution of the Pig job
> from within it.
>
> Thanks
>
> On Mon, Sep 12, 2016 at 1:44 AM, Peter Cseh <ge...@cloudera.com> wrote:
>
>> Hi,
>>
>> Can you submit jobs into the specified queues outside Oozie?
>> Is there any reason Pig Action
>> <https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#
>> a3.2.3_Pig_Action>
>> can't
>> be used instead of Shell Action to submit Pig jobs?
>>
>> Thanks,
>> Peter
>>
>>
>>
>> On Sat, Sep 10, 2016 at 7:37 PM, Bijoy Deb <bi...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I have configured 2 new queues 'launcher' and 'mapreduce' apart from
>>> 'default' in Capacity scheduler in yarn-site.
>>> Now, I am trying to submit a Pig job via *Oozie Shell action* into those
>>> queues queues such that the oozie launcher job goes into 'launcher' and
>> pig
>>> job into 'mapreduce' queue.On submitting the oozie workflow I can see
>> that
>>> my launcher vjob successfully runs in 'launcher' queue,but the
>>> Pig/mapreduce job is still running in default queue.
>>>
>>> Below is the snippet that I added to my workflow.xml for above:
>>>
>>> *<configuration>*
>>>
>>> *<property>*
>>>
>>> *<name>oozie.launcher.mapreduce.job.queuename</name>*
>>>
>>> *<value>launcher</value>*
>>>
>>> *</property>*
>>>
>>> *<property>*
>>>
>>> *<name>mapreduce.job.queuename</name>*
>>>
>>> *<value>mapreduce</value>*
>>> *</property>*
>>> *</configuration>*
>>>
>>> I also tried using 'mapred.job.queue.name' instead of
>>> 'mapreduce.job.queuename' above and also by passing it as
>>> -Dmapreduce.job.queuename=mapreduce while submitting the oozie job via
>>> Oozie cli.But in every case the Pig/mapreduce job goes into 'default'
>>> queue.
>>>
>>> Am I doing anything wrong here or is it that Oozie Shell action doesn't
>>> support 'mapreduce.job.quenename specification?
>>>
>>> I am using MR2, Hadoop version 2.7.1, oozie version 4.2.0.
>>>
>>> Any help would be really appreciated.
>>>
>>> Thanks
>>>
>>
>>
>>
>> --
>> Peter Cseh
>> Software Engineer
>> <http://www.cloudera.com>
>>
Re: Unable to configure custom mapreduce.job.queuename in Oozie shell action
Posted by Bijoy Deb <bi...@gmail.com>.
Hi,
When I submit a job using Pig command,I am able to submit it to the custom
queue using 'set mapreduce.job.queuename <name_of_queue>' command inside
Pig script.It seems the problem is when I specify the queuename in
<configuration> tag via Oozie shell action.
We cannot use the Pig script directly and use shell action instead as we
have some custom operations such as logging,auditing,etc. that we want to
perform via the shell script during the course of execution of the Pig job
from within it.
Thanks
On Mon, Sep 12, 2016 at 1:44 AM, Peter Cseh <ge...@cloudera.com> wrote:
> Hi,
>
> Can you submit jobs into the specified queues outside Oozie?
> Is there any reason Pig Action
> <https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#
> a3.2.3_Pig_Action>
> can't
> be used instead of Shell Action to submit Pig jobs?
>
> Thanks,
> Peter
>
>
>
> On Sat, Sep 10, 2016 at 7:37 PM, Bijoy Deb <bi...@gmail.com>
> wrote:
>
> > Hi,
> > I have configured 2 new queues 'launcher' and 'mapreduce' apart from
> > 'default' in Capacity scheduler in yarn-site.
> > Now, I am trying to submit a Pig job via *Oozie Shell action* into those
> > queues queues such that the oozie launcher job goes into 'launcher' and
> pig
> > job into 'mapreduce' queue.On submitting the oozie workflow I can see
> that
> > my launcher vjob successfully runs in 'launcher' queue,but the
> > Pig/mapreduce job is still running in default queue.
> >
> > Below is the snippet that I added to my workflow.xml for above:
> >
> > *<configuration>*
> >
> > *<property>*
> >
> > *<name>oozie.launcher.mapreduce.job.queuename</name>*
> >
> > *<value>launcher</value>*
> >
> > *</property>*
> >
> > *<property>*
> >
> > *<name>mapreduce.job.queuename</name>*
> >
> > *<value>mapreduce</value>*
> > *</property>*
> > *</configuration>*
> >
> > I also tried using 'mapred.job.queue.name' instead of
> > 'mapreduce.job.queuename' above and also by passing it as
> > -Dmapreduce.job.queuename=mapreduce while submitting the oozie job via
> > Oozie cli.But in every case the Pig/mapreduce job goes into 'default'
> > queue.
> >
> > Am I doing anything wrong here or is it that Oozie Shell action doesn't
> > support 'mapreduce.job.quenename specification?
> >
> > I am using MR2, Hadoop version 2.7.1, oozie version 4.2.0.
> >
> > Any help would be really appreciated.
> >
> > Thanks
> >
>
>
>
> --
> Peter Cseh
> Software Engineer
> <http://www.cloudera.com>
>
Re: Unable to configure custom mapreduce.job.queuename in Oozie shell action
Posted by Peter Cseh <ge...@cloudera.com>.
Hi,
Can you submit jobs into the specified queues outside Oozie?
Is there any reason Pig Action
<https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.2.3_Pig_Action>
can't
be used instead of Shell Action to submit Pig jobs?
Thanks,
Peter
On Sat, Sep 10, 2016 at 7:37 PM, Bijoy Deb <bi...@gmail.com>
wrote:
> Hi,
> I have configured 2 new queues 'launcher' and 'mapreduce' apart from
> 'default' in Capacity scheduler in yarn-site.
> Now, I am trying to submit a Pig job via *Oozie Shell action* into those
> queues queues such that the oozie launcher job goes into 'launcher' and pig
> job into 'mapreduce' queue.On submitting the oozie workflow I can see that
> my launcher vjob successfully runs in 'launcher' queue,but the
> Pig/mapreduce job is still running in default queue.
>
> Below is the snippet that I added to my workflow.xml for above:
>
> *<configuration>*
>
> *<property>*
>
> *<name>oozie.launcher.mapreduce.job.queuename</name>*
>
> *<value>launcher</value>*
>
> *</property>*
>
> *<property>*
>
> *<name>mapreduce.job.queuename</name>*
>
> *<value>mapreduce</value>*
> *</property>*
> *</configuration>*
>
> I also tried using 'mapred.job.queue.name' instead of
> 'mapreduce.job.queuename' above and also by passing it as
> -Dmapreduce.job.queuename=mapreduce while submitting the oozie job via
> Oozie cli.But in every case the Pig/mapreduce job goes into 'default'
> queue.
>
> Am I doing anything wrong here or is it that Oozie Shell action doesn't
> support 'mapreduce.job.quenename specification?
>
> I am using MR2, Hadoop version 2.7.1, oozie version 4.2.0.
>
> Any help would be really appreciated.
>
> Thanks
>
--
Peter Cseh
Software Engineer
<http://www.cloudera.com>