You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Bhagaban Khatai <em...@gmail.com> on 2015/04/01 07:56:01 UTC
Oozie Load distribution
Hi,
Can someone explain me, if oozie doing the load distribution in the
cluster. I have to submit 100 shell script using fork, will it equally
distribute across all nodes.
Need a little depth knowledge on oozie load balancing.
Please help
Re: Oozie Load distribution
Posted by Bhagaban Khatai <em...@gmail.com>.
Thanks
On Wed, Apr 1, 2015 at 12:12 PM, Satbeer Lamba <sa...@infoworks.io> wrote:
> Oozie will execute each script as a 1 Mapper 0 reducer job. Therefore you
> can think of it as a normal mapper job with 100 mapper tasks and hence
> these tasks will be distributed across cluster accordingly.
>
> In case you are looking for something that would run a set of commands
> across all nodes in your cluster, you can see if parallel-ssh
> <https://github.com/pkittenis/parallel-ssh> suits your requirements.
>
> Thanks
> Satbeer
>
> On Wed, Apr 1, 2015 at 11:52 AM, Bhagaban Khatai <email.bhagaban@gmail.com
> >
> wrote:
>
> > Our requirement is 100 script would run from different node via oozie
> (can
> > be a shell action or SSH action), in this scenario what best option you
> can
> > advice. so that it will distribute across all nodes and do some load
> > balancing. I doutb if many script will run in one perticular node then it
> > might choke the system.
> >
> > Your help would be greatly appriciated.
> >
> > On Wed, Apr 1, 2015 at 11:32 AM, Alejandro Abdelnur <tu...@gmail.com>
> > wrote:
> >
> > > oozie simply submits the job to run the script to the cluster, it is up
> > to
> > > the cluster scheduler what nodes are allocated.
> > >
> > > hth
> > >
> > > Alejandro
> > >
> > > > On Mar 31, 2015, at 22:56, Bhagaban Khatai <email.bhagaban@gmail.com
> >
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > Can someone explain me, if oozie doing the load distribution in the
> > > > cluster. I have to submit 100 shell script using fork, will it
> equally
> > > > distribute across all nodes.
> > > >
> > > > Need a little depth knowledge on oozie load balancing.
> > > >
> > > > Please help
> > >
> >
>
Re: Oozie Load distribution
Posted by Satbeer Lamba <sa...@infoworks.io>.
Oozie will execute each script as a 1 Mapper 0 reducer job. Therefore you
can think of it as a normal mapper job with 100 mapper tasks and hence
these tasks will be distributed across cluster accordingly.
In case you are looking for something that would run a set of commands
across all nodes in your cluster, you can see if parallel-ssh
<https://github.com/pkittenis/parallel-ssh> suits your requirements.
Thanks
Satbeer
On Wed, Apr 1, 2015 at 11:52 AM, Bhagaban Khatai <em...@gmail.com>
wrote:
> Our requirement is 100 script would run from different node via oozie (can
> be a shell action or SSH action), in this scenario what best option you can
> advice. so that it will distribute across all nodes and do some load
> balancing. I doutb if many script will run in one perticular node then it
> might choke the system.
>
> Your help would be greatly appriciated.
>
> On Wed, Apr 1, 2015 at 11:32 AM, Alejandro Abdelnur <tu...@gmail.com>
> wrote:
>
> > oozie simply submits the job to run the script to the cluster, it is up
> to
> > the cluster scheduler what nodes are allocated.
> >
> > hth
> >
> > Alejandro
> >
> > > On Mar 31, 2015, at 22:56, Bhagaban Khatai <em...@gmail.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > Can someone explain me, if oozie doing the load distribution in the
> > > cluster. I have to submit 100 shell script using fork, will it equally
> > > distribute across all nodes.
> > >
> > > Need a little depth knowledge on oozie load balancing.
> > >
> > > Please help
> >
>
Re: Oozie Load distribution
Posted by Bhagaban Khatai <em...@gmail.com>.
Our requirement is 100 script would run from different node via oozie (can
be a shell action or SSH action), in this scenario what best option you can
advice. so that it will distribute across all nodes and do some load
balancing. I doutb if many script will run in one perticular node then it
might choke the system.
Your help would be greatly appriciated.
On Wed, Apr 1, 2015 at 11:32 AM, Alejandro Abdelnur <tu...@gmail.com>
wrote:
> oozie simply submits the job to run the script to the cluster, it is up to
> the cluster scheduler what nodes are allocated.
>
> hth
>
> Alejandro
>
> > On Mar 31, 2015, at 22:56, Bhagaban Khatai <em...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > Can someone explain me, if oozie doing the load distribution in the
> > cluster. I have to submit 100 shell script using fork, will it equally
> > distribute across all nodes.
> >
> > Need a little depth knowledge on oozie load balancing.
> >
> > Please help
>
Re: Oozie Load distribution
Posted by Alejandro Abdelnur <tu...@gmail.com>.
oozie simply submits the job to run the script to the cluster, it is up to the cluster scheduler what nodes are allocated.
hth
Alejandro
> On Mar 31, 2015, at 22:56, Bhagaban Khatai <em...@gmail.com> wrote:
>
> Hi,
>
> Can someone explain me, if oozie doing the load distribution in the
> cluster. I have to submit 100 shell script using fork, will it equally
> distribute across all nodes.
>
> Need a little depth knowledge on oozie load balancing.
>
> Please help