You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Vincent Russell <vi...@gmail.com> on 2015/07/22 18:46:13 UTC
Limiting the current workflows that run concurrenlty
Hello,
I am using oozie version 4.1.0 and I am trying to limit the number of
workflows (across the entire oozie application) that can be run at the same
time.
I have tried setting what I thought was the appropriate property,
oozie.service.CallableQueueService.callable.concurrency, but it seems like
it is being ignored.
Thanks,
Re: Limiting the current workflows that run concurrenlty
Posted by Vincent Russell <vi...@gmail.com>.
That's a good suggestion.
Thank you,
On Tue, Jul 28, 2015 at 6:11 PM, Jagat Singh <ja...@gmail.com> wrote:
> You can push launcher jobs to some different queue and limit that to
> required resources.
>
> oozie.launcher.mapred.job.queue.name
>
>
>
> On Thu, Jul 23, 2015 at 3:33 AM, Vincent Russell <
> vincent.russell@gmail.com>
> wrote:
>
> > Thank you Robert for you quick response.
> >
> > My use case is to prevent oozie from allowing a user to fire up a bunch
> of
> > concurrent jobs that could reduce performance of the cluster.
> >
> > Thanks,
> >
> > On Wed, Jul 22, 2015 at 1:17 PM, Robert Kanter <rk...@cloudera.com>
> > wrote:
> >
> > > Hi Vincent,
> > >
> > > There isn't currently a way to limit the number of workflows run across
> > the
> > > entire system. You can limit the number of concurrent workflows
> launched
> > > by a single Coordinator though. I suppose you could do something
> similar
> > > to what you want by putting all of the Launcher Jobs into the same MR
> > > queue, so that you can limit the number of concurrent actions running
> in
> > > the cluster, but you'd still have no limit on the number of workflows.
> > > What is your use case for this?
> > >
> > > oozie.service.CallableQueueService.callable.concurrency controls the
> > > concurrency of some internal processing that Oozie does. Basically,
> the
> > > way Oozie works is that it has a queue of (internal) commands that it
> > > runs. When something needs to be done, it gets added to this queue.
> So
> > > that property simply controls how many of the same type of command can
> be
> > > processed concurrently.
> > >
> > > - Robert
> > >
> > > On Wed, Jul 22, 2015 at 9:46 AM, Vincent Russell <
> > > vincent.russell@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I am using oozie version 4.1.0 and I am trying to limit the number of
> > > > workflows (across the entire oozie application) that can be run at
> the
> > > same
> > > > time.
> > > >
> > > > I have tried setting what I thought was the appropriate property,
> > > > oozie.service.CallableQueueService.callable.concurrency, but it seems
> > > like
> > > > it is being ignored.
> > > >
> > > > Thanks,
> > > >
> > >
> >
>
Re: Limiting the current workflows that run concurrenlty
Posted by Jagat Singh <ja...@gmail.com>.
You can push launcher jobs to some different queue and limit that to
required resources.
oozie.launcher.mapred.job.queue.name
On Thu, Jul 23, 2015 at 3:33 AM, Vincent Russell <vi...@gmail.com>
wrote:
> Thank you Robert for you quick response.
>
> My use case is to prevent oozie from allowing a user to fire up a bunch of
> concurrent jobs that could reduce performance of the cluster.
>
> Thanks,
>
> On Wed, Jul 22, 2015 at 1:17 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > Hi Vincent,
> >
> > There isn't currently a way to limit the number of workflows run across
> the
> > entire system. You can limit the number of concurrent workflows launched
> > by a single Coordinator though. I suppose you could do something similar
> > to what you want by putting all of the Launcher Jobs into the same MR
> > queue, so that you can limit the number of concurrent actions running in
> > the cluster, but you'd still have no limit on the number of workflows.
> > What is your use case for this?
> >
> > oozie.service.CallableQueueService.callable.concurrency controls the
> > concurrency of some internal processing that Oozie does. Basically, the
> > way Oozie works is that it has a queue of (internal) commands that it
> > runs. When something needs to be done, it gets added to this queue. So
> > that property simply controls how many of the same type of command can be
> > processed concurrently.
> >
> > - Robert
> >
> > On Wed, Jul 22, 2015 at 9:46 AM, Vincent Russell <
> > vincent.russell@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I am using oozie version 4.1.0 and I am trying to limit the number of
> > > workflows (across the entire oozie application) that can be run at the
> > same
> > > time.
> > >
> > > I have tried setting what I thought was the appropriate property,
> > > oozie.service.CallableQueueService.callable.concurrency, but it seems
> > like
> > > it is being ignored.
> > >
> > > Thanks,
> > >
> >
>
Re: Limiting the current workflows that run concurrenlty
Posted by Vincent Russell <vi...@gmail.com>.
Thank you Robert for you quick response.
My use case is to prevent oozie from allowing a user to fire up a bunch of
concurrent jobs that could reduce performance of the cluster.
Thanks,
On Wed, Jul 22, 2015 at 1:17 PM, Robert Kanter <rk...@cloudera.com> wrote:
> Hi Vincent,
>
> There isn't currently a way to limit the number of workflows run across the
> entire system. You can limit the number of concurrent workflows launched
> by a single Coordinator though. I suppose you could do something similar
> to what you want by putting all of the Launcher Jobs into the same MR
> queue, so that you can limit the number of concurrent actions running in
> the cluster, but you'd still have no limit on the number of workflows.
> What is your use case for this?
>
> oozie.service.CallableQueueService.callable.concurrency controls the
> concurrency of some internal processing that Oozie does. Basically, the
> way Oozie works is that it has a queue of (internal) commands that it
> runs. When something needs to be done, it gets added to this queue. So
> that property simply controls how many of the same type of command can be
> processed concurrently.
>
> - Robert
>
> On Wed, Jul 22, 2015 at 9:46 AM, Vincent Russell <
> vincent.russell@gmail.com>
> wrote:
>
> > Hello,
> >
> > I am using oozie version 4.1.0 and I am trying to limit the number of
> > workflows (across the entire oozie application) that can be run at the
> same
> > time.
> >
> > I have tried setting what I thought was the appropriate property,
> > oozie.service.CallableQueueService.callable.concurrency, but it seems
> like
> > it is being ignored.
> >
> > Thanks,
> >
>
Re: Limiting the current workflows that run concurrenlty
Posted by Robert Kanter <rk...@cloudera.com>.
Hi Vincent,
There isn't currently a way to limit the number of workflows run across the
entire system. You can limit the number of concurrent workflows launched
by a single Coordinator though. I suppose you could do something similar
to what you want by putting all of the Launcher Jobs into the same MR
queue, so that you can limit the number of concurrent actions running in
the cluster, but you'd still have no limit on the number of workflows.
What is your use case for this?
oozie.service.CallableQueueService.callable.concurrency controls the
concurrency of some internal processing that Oozie does. Basically, the
way Oozie works is that it has a queue of (internal) commands that it
runs. When something needs to be done, it gets added to this queue. So
that property simply controls how many of the same type of command can be
processed concurrently.
- Robert
On Wed, Jul 22, 2015 at 9:46 AM, Vincent Russell <vi...@gmail.com>
wrote:
> Hello,
>
> I am using oozie version 4.1.0 and I am trying to limit the number of
> workflows (across the entire oozie application) that can be run at the same
> time.
>
> I have tried setting what I thought was the appropriate property,
> oozie.service.CallableQueueService.callable.concurrency, but it seems like
> it is being ignored.
>
> Thanks,
>