You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Prakhar Srivastava <pr...@gmail.com> on 2012/03/13 18:55:13 UTC
Non hadoop job execution in oozie
Hi,
Oozie has a action node called ssh. Can someone throw some light on how ssh
action is executed.
I found out that Java Action is executed as single mapper only job on one
of the hadoop node. I want to just execute a python script on one of the
node(not on hadoop cluster) of my choice. How can I accomplish this ??
Prakhar
Re: Non hadoop job execution in oozie
Posted by Prakhar Srivastava <pr...@gmail.com>.
Hi Harsh,
Thanks for the clarification.
Prakhar
On Wed, Mar 14, 2012 at 3:27 PM, Harsh J <ha...@cloudera.com> wrote:
> Prakhar,
>
> Mohammad's second point talks of a Shell (simple command) action,
> which is different from an SSH (remote login) action. The former will
> run as any other JavaAction, within the MR cluster. The latter will
> make an SSH logon to a specified destination if it is able to.
>
> On Wed, Mar 14, 2012 at 3:22 PM, Prakhar Srivastava
> <pr...@gmail.com> wrote:
> > Hi Mohammad,
> >
> > Thanks for the response.
> >
> > As far as I have read, the SSH action requires one to *specify the remote
> > host* at which the command/script will be executing.(assuming that
> > command/scripts are present on the specified host machine).
> >
> > Your second point says about a restriction that the command will be
> *executed
> > on the hadoop cluster*. I think I am missing something because SSH action
> > and this restriction contradicts each other.
> >
> > Prakhar
> >
> > On Wed, Mar 14, 2012 at 2:26 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
> >
> >> Hi Prakhar,
> >> Adding on Alejandro's comments.
> >> * SSH action was deprecated in earlier version. But this restriction has
> >> gone in the latest releases. SSH action will be there. I think it is the
> >> choice for your use case.
> >>
> >> * Just FYI, in 3.2+, oozie provides Shell action to run any shell
> command
> >> including perl/ruby/php provided they are installed in the compute node.
> >> Another restriction: the command will be executed in the Hadoop cluster
> >> which might not be your use case.
> >>
> >> Regards,
> >> Mohammad
> >>
> >>
> >> ________________________________
> >> From: Prakhar Srivastava <pr...@gmail.com>
> >> To: oozie-users@incubator.apache.org
> >> Sent: Tuesday, March 13, 2012 10:44 PM
> >> Subject: Re: Non hadoop job execution in oozie
> >>
> >> Hi Alejandro,
> >> Thanks for the response.
> >>
> >> I found out that ssh action is being deprecated in Oozie. If a user
> wants
> >> to launch a python/ruby script on a specified node, how can this be
> >> achieved in oozie. For instance, I wish to pull some data from the
> network,
> >> I would not like to execute this script on a Hadoop cluster.
> >>
> >> Oozie seems tightly coupled with the Hadoop cluster. Can it be extended
> for
> >> running ad-hoc jobs in workflow on non hadoop environment??
> >>
> >> Prakhar
> >>
> >> On Wed, Mar 14, 2012 at 4:27 AM, Alejandro Abdelnur <tucu@cloudera.com
> >> >wrote:
> >>
> >> > Prakhar,
> >> >
> >> > As you suspect, the SSH action is a the target SSH host specified in
> the
> >> > SSH action, it does not involved a Hadoop node.
> >> >
> >> > You have to create an SSH key for oozie and set the public key in the
> >> > target SSH host.
> >> >
> >> > hope this helps
> >> >
> >> > Alejandro
> >> >
> >> > On Tue, Mar 13, 2012 at 10:55 AM, Prakhar Srivastava
> >> > <pr...@gmail.com>wrote:
> >> >
> >> > > Hi,
> >> > > Oozie has a action node called ssh. Can someone throw some light on
> how
> >> > ssh
> >> > > action is executed.
> >> > >
> >> > > I found out that Java Action is executed as single mapper only job
> on
> >> one
> >> > > of the hadoop node. I want to just execute a python script on one of
> >> the
> >> > > node(not on hadoop cluster) of my choice. How can I accomplish this
> ??
> >> > >
> >> > > Prakhar
> >> > >
> >> >
> >>
>
>
>
> --
> Harsh J
>
Re: Non hadoop job execution in oozie
Posted by Harsh J <ha...@cloudera.com>.
Prakhar,
Mohammad's second point talks of a Shell (simple command) action,
which is different from an SSH (remote login) action. The former will
run as any other JavaAction, within the MR cluster. The latter will
make an SSH logon to a specified destination if it is able to.
On Wed, Mar 14, 2012 at 3:22 PM, Prakhar Srivastava
<pr...@gmail.com> wrote:
> Hi Mohammad,
>
> Thanks for the response.
>
> As far as I have read, the SSH action requires one to *specify the remote
> host* at which the command/script will be executing.(assuming that
> command/scripts are present on the specified host machine).
>
> Your second point says about a restriction that the command will be *executed
> on the hadoop cluster*. I think I am missing something because SSH action
> and this restriction contradicts each other.
>
> Prakhar
>
> On Wed, Mar 14, 2012 at 2:26 PM, Mohammad Islam <mi...@yahoo.com> wrote:
>
>> Hi Prakhar,
>> Adding on Alejandro's comments.
>> * SSH action was deprecated in earlier version. But this restriction has
>> gone in the latest releases. SSH action will be there. I think it is the
>> choice for your use case.
>>
>> * Just FYI, in 3.2+, oozie provides Shell action to run any shell command
>> including perl/ruby/php provided they are installed in the compute node.
>> Another restriction: the command will be executed in the Hadoop cluster
>> which might not be your use case.
>>
>> Regards,
>> Mohammad
>>
>>
>> ________________________________
>> From: Prakhar Srivastava <pr...@gmail.com>
>> To: oozie-users@incubator.apache.org
>> Sent: Tuesday, March 13, 2012 10:44 PM
>> Subject: Re: Non hadoop job execution in oozie
>>
>> Hi Alejandro,
>> Thanks for the response.
>>
>> I found out that ssh action is being deprecated in Oozie. If a user wants
>> to launch a python/ruby script on a specified node, how can this be
>> achieved in oozie. For instance, I wish to pull some data from the network,
>> I would not like to execute this script on a Hadoop cluster.
>>
>> Oozie seems tightly coupled with the Hadoop cluster. Can it be extended for
>> running ad-hoc jobs in workflow on non hadoop environment??
>>
>> Prakhar
>>
>> On Wed, Mar 14, 2012 at 4:27 AM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>>
>> > Prakhar,
>> >
>> > As you suspect, the SSH action is a the target SSH host specified in the
>> > SSH action, it does not involved a Hadoop node.
>> >
>> > You have to create an SSH key for oozie and set the public key in the
>> > target SSH host.
>> >
>> > hope this helps
>> >
>> > Alejandro
>> >
>> > On Tue, Mar 13, 2012 at 10:55 AM, Prakhar Srivastava
>> > <pr...@gmail.com>wrote:
>> >
>> > > Hi,
>> > > Oozie has a action node called ssh. Can someone throw some light on how
>> > ssh
>> > > action is executed.
>> > >
>> > > I found out that Java Action is executed as single mapper only job on
>> one
>> > > of the hadoop node. I want to just execute a python script on one of
>> the
>> > > node(not on hadoop cluster) of my choice. How can I accomplish this ??
>> > >
>> > > Prakhar
>> > >
>> >
>>
--
Harsh J
Re: Non hadoop job execution in oozie
Posted by Prakhar Srivastava <pr...@gmail.com>.
Hi Mohammad,
Thanks for the response.
As far as I have read, the SSH action requires one to *specify the remote
host* at which the command/script will be executing.(assuming that
command/scripts are present on the specified host machine).
Your second point says about a restriction that the command will be *executed
on the hadoop cluster*. I think I am missing something because SSH action
and this restriction contradicts each other.
Prakhar
On Wed, Mar 14, 2012 at 2:26 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> Hi Prakhar,
> Adding on Alejandro's comments.
> * SSH action was deprecated in earlier version. But this restriction has
> gone in the latest releases. SSH action will be there. I think it is the
> choice for your use case.
>
> * Just FYI, in 3.2+, oozie provides Shell action to run any shell command
> including perl/ruby/php provided they are installed in the compute node.
> Another restriction: the command will be executed in the Hadoop cluster
> which might not be your use case.
>
> Regards,
> Mohammad
>
>
> ________________________________
> From: Prakhar Srivastava <pr...@gmail.com>
> To: oozie-users@incubator.apache.org
> Sent: Tuesday, March 13, 2012 10:44 PM
> Subject: Re: Non hadoop job execution in oozie
>
> Hi Alejandro,
> Thanks for the response.
>
> I found out that ssh action is being deprecated in Oozie. If a user wants
> to launch a python/ruby script on a specified node, how can this be
> achieved in oozie. For instance, I wish to pull some data from the network,
> I would not like to execute this script on a Hadoop cluster.
>
> Oozie seems tightly coupled with the Hadoop cluster. Can it be extended for
> running ad-hoc jobs in workflow on non hadoop environment??
>
> Prakhar
>
> On Wed, Mar 14, 2012 at 4:27 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Prakhar,
> >
> > As you suspect, the SSH action is a the target SSH host specified in the
> > SSH action, it does not involved a Hadoop node.
> >
> > You have to create an SSH key for oozie and set the public key in the
> > target SSH host.
> >
> > hope this helps
> >
> > Alejandro
> >
> > On Tue, Mar 13, 2012 at 10:55 AM, Prakhar Srivastava
> > <pr...@gmail.com>wrote:
> >
> > > Hi,
> > > Oozie has a action node called ssh. Can someone throw some light on how
> > ssh
> > > action is executed.
> > >
> > > I found out that Java Action is executed as single mapper only job on
> one
> > > of the hadoop node. I want to just execute a python script on one of
> the
> > > node(not on hadoop cluster) of my choice. How can I accomplish this ??
> > >
> > > Prakhar
> > >
> >
>
Re: Non hadoop job execution in oozie
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Prakhar,
Adding on Alejandro's comments.
* SSH action was deprecated in earlier version. But this restriction has gone in the latest releases. SSH action will be there. I think it is the choice for your use case.
* Just FYI, in 3.2+, oozie provides Shell action to run any shell command including perl/ruby/php provided they are installed in the compute node. Another restriction: the command will be executed in the Hadoop cluster which might not be your use case.
Regards,
Mohammad
________________________________
From: Prakhar Srivastava <pr...@gmail.com>
To: oozie-users@incubator.apache.org
Sent: Tuesday, March 13, 2012 10:44 PM
Subject: Re: Non hadoop job execution in oozie
Hi Alejandro,
Thanks for the response.
I found out that ssh action is being deprecated in Oozie. If a user wants
to launch a python/ruby script on a specified node, how can this be
achieved in oozie. For instance, I wish to pull some data from the network,
I would not like to execute this script on a Hadoop cluster.
Oozie seems tightly coupled with the Hadoop cluster. Can it be extended for
running ad-hoc jobs in workflow on non hadoop environment??
Prakhar
On Wed, Mar 14, 2012 at 4:27 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Prakhar,
>
> As you suspect, the SSH action is a the target SSH host specified in the
> SSH action, it does not involved a Hadoop node.
>
> You have to create an SSH key for oozie and set the public key in the
> target SSH host.
>
> hope this helps
>
> Alejandro
>
> On Tue, Mar 13, 2012 at 10:55 AM, Prakhar Srivastava
> <pr...@gmail.com>wrote:
>
> > Hi,
> > Oozie has a action node called ssh. Can someone throw some light on how
> ssh
> > action is executed.
> >
> > I found out that Java Action is executed as single mapper only job on one
> > of the hadoop node. I want to just execute a python script on one of the
> > node(not on hadoop cluster) of my choice. How can I accomplish this ??
> >
> > Prakhar
> >
>
Re: Non hadoop job execution in oozie
Posted by Prakhar Srivastava <pr...@gmail.com>.
Hi Alejandro,
Thanks for the response.
I found out that ssh action is being deprecated in Oozie. If a user wants
to launch a python/ruby script on a specified node, how can this be
achieved in oozie. For instance, I wish to pull some data from the network,
I would not like to execute this script on a Hadoop cluster.
Oozie seems tightly coupled with the Hadoop cluster. Can it be extended for
running ad-hoc jobs in workflow on non hadoop environment??
Prakhar
On Wed, Mar 14, 2012 at 4:27 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Prakhar,
>
> As you suspect, the SSH action is a the target SSH host specified in the
> SSH action, it does not involved a Hadoop node.
>
> You have to create an SSH key for oozie and set the public key in the
> target SSH host.
>
> hope this helps
>
> Alejandro
>
> On Tue, Mar 13, 2012 at 10:55 AM, Prakhar Srivastava
> <pr...@gmail.com>wrote:
>
> > Hi,
> > Oozie has a action node called ssh. Can someone throw some light on how
> ssh
> > action is executed.
> >
> > I found out that Java Action is executed as single mapper only job on one
> > of the hadoop node. I want to just execute a python script on one of the
> > node(not on hadoop cluster) of my choice. How can I accomplish this ??
> >
> > Prakhar
> >
>
Re: Non hadoop job execution in oozie
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Prakhar,
As you suspect, the SSH action is a the target SSH host specified in the
SSH action, it does not involved a Hadoop node.
You have to create an SSH key for oozie and set the public key in the
target SSH host.
hope this helps
Alejandro
On Tue, Mar 13, 2012 at 10:55 AM, Prakhar Srivastava
<pr...@gmail.com>wrote:
> Hi,
> Oozie has a action node called ssh. Can someone throw some light on how ssh
> action is executed.
>
> I found out that Java Action is executed as single mapper only job on one
> of the hadoop node. I want to just execute a python script on one of the
> node(not on hadoop cluster) of my choice. How can I accomplish this ??
>
> Prakhar
>