You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Derwin Chiang <de...@yahoo-inc.com> on 2011/12/05 12:06:50 UTC
Is it possible to run multiple grid jobs through a script
Hi, All
I am doing migration for running cron job on grid,
Originally, we wrote one script and run the script every day automatically, there are some commands for triggering hadoop/pig jobs in this script.
Since we can run script on oozie through some java action,
Is it possible that we directly run this script through this way ?
I have taken some try, it seems that it can run this script, but no result output.
Does anyone have this experience ? Or do you know how to do this?
Thanks
Derwin
Re: Is it possible to run multiple grid jobs through a script
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Derwin,
So nothing is printed for command "print `pig -Dmapred.job.queue.name=XXXXX MY_PIG_Script.pig`;".
Can you play with this line.
One option cold be:
Redirect the output to a tmp file and then "cat" the tmp file.
Another option: output=`pig ...`. then "print $output"
Regards,
Mohammad
________________________________
From: Derwin Chiang <de...@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>; Mohammad Islam <mi...@yahoo.com>
Sent: Tuesday, December 6, 2011 1:32 AM
Subject: RE: Is it possible to run multiple grid jobs through a script
Hi, Mohammad
My java code would be like the "Java-Main Action with Script support" section in https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
And for my perl script, I just run some simple line like this:
My_script.pl
=========================================================
#!/usr/local/bin/perl
print "start from here\n";
print `pig -Dmapred.job.queue.name=XXXXX MY_PIG_Script.pig`;
print "finished the script";
=========================================================
And without oozie, I could run this script with type command "perl ./My_script.pl" in command line
But through oozie, although I can see the "start from here" & "finished the script" print out, but I don't see anything related to the pig script, neither the output from the pig script.
So I guess maybe this way is not supported :-(
Thanks
Derwin
-----Original Message-----
From: Mohammad Islam [mailto:mislam77@yahoo.com]
Sent: Tuesday, December 06, 2011 4:01 PM
To: oozie-users@incubator.apache.org
Subject: Re: Is it possible to run multiple grid jobs through a script
Hi Derwin,
As Alejandro mention, it is not a good practice.
I'm wandering what is the problem you are getting.
Don't you see your output in the Java Astion Launcher log. If it is missing from there, I would like to see the java code that execute the script.
Regards,
Mohammad
________________________________
From: Derwin Chiang <de...@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
Sent: Monday, December 5, 2011 7:47 PM
Subject: RE: Is it possible to run multiple grid jobs through a script
Thanks Alejandro
I know this idea is not a good practice.
But why we need this is just because that these implementation were came from other team,
And we don't have enough time to figure out everything in these implementation for rewriting the workflow in oozie.
So we wonder know can we simply migration it to oozie with little effort.
But if it's not supported by oozie, hmmm, OK, at least we know we would have a long way to go
Thanks.
Derwin
-----Original Message-----
From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
Sent: Tuesday, December 06, 2011 1:38 AM
To: oozie-users@incubator.apache.org
Subject: Re: Is it possible to run multiple grid jobs through a script
Derwin,
Doing as you describe it means it will be quite difficult for you track
where your jobs are and their status.
I'd recommend wiring all your hadoop/pig jobs in workflow and then using a
coordinator to trigger them.
Regards.
Alejandro
On Mon, Dec 5, 2011 at 3:06 AM, Derwin Chiang <de...@yahoo-inc.com> wrote:
> Hi, All
>
> I am doing migration for running cron job on grid,
> Originally, we wrote one script and run the script every day
> automatically, there are some commands for triggering hadoop/pig jobs in
> this script.
> Since we can run script on oozie through some java action,
> Is it possible that we directly run this script through this way ?
> I have taken some try, it seems that it can run this script, but no
> result output.
> Does anyone have this experience ? Or do you know how to do this?
> Thanks
>
> Derwin
>
RE: Is it possible to run multiple grid jobs through a script
Posted by Derwin Chiang <de...@yahoo-inc.com>.
Hi, Mohammad
My java code would be like the "Java-Main Action with Script support" section in https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
And for my perl script, I just run some simple line like this:
My_script.pl
=========================================================
#!/usr/local/bin/perl
print "start from here\n";
print `pig -Dmapred.job.queue.name=XXXXX MY_PIG_Script.pig`;
print "finished the script";
=========================================================
And without oozie, I could run this script with type command "perl ./My_script.pl" in command line
But through oozie, although I can see the "start from here" & "finished the script" print out, but I don't see anything related to the pig script, neither the output from the pig script.
So I guess maybe this way is not supported :-(
Thanks
Derwin
-----Original Message-----
From: Mohammad Islam [mailto:mislam77@yahoo.com]
Sent: Tuesday, December 06, 2011 4:01 PM
To: oozie-users@incubator.apache.org
Subject: Re: Is it possible to run multiple grid jobs through a script
Hi Derwin,
As Alejandro mention, it is not a good practice.
I'm wandering what is the problem you are getting.
Don't you see your output in the Java Astion Launcher log. If it is missing from there, I would like to see the java code that execute the script.
Regards,
Mohammad
________________________________
From: Derwin Chiang <de...@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
Sent: Monday, December 5, 2011 7:47 PM
Subject: RE: Is it possible to run multiple grid jobs through a script
Thanks Alejandro
I know this idea is not a good practice.
But why we need this is just because that these implementation were came from other team,
And we don't have enough time to figure out everything in these implementation for rewriting the workflow in oozie.
So we wonder know can we simply migration it to oozie with little effort.
But if it's not supported by oozie, hmmm, OK, at least we know we would have a long way to go
Thanks.
Derwin
-----Original Message-----
From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
Sent: Tuesday, December 06, 2011 1:38 AM
To: oozie-users@incubator.apache.org
Subject: Re: Is it possible to run multiple grid jobs through a script
Derwin,
Doing as you describe it means it will be quite difficult for you track
where your jobs are and their status.
I'd recommend wiring all your hadoop/pig jobs in workflow and then using a
coordinator to trigger them.
Regards.
Alejandro
On Mon, Dec 5, 2011 at 3:06 AM, Derwin Chiang <de...@yahoo-inc.com> wrote:
> Hi, All
>
> I am doing migration for running cron job on grid,
> Originally, we wrote one script and run the script every day
> automatically, there are some commands for triggering hadoop/pig jobs in
> this script.
> Since we can run script on oozie through some java action,
> Is it possible that we directly run this script through this way ?
> I have taken some try, it seems that it can run this script, but no
> result output.
> Does anyone have this experience ? Or do you know how to do this?
> Thanks
>
> Derwin
>
Re: Is it possible to run multiple grid jobs through a script
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Derwin,
As Alejandro mention, it is not a good practice.
I'm wandering what is the problem you are getting.
Don't you see your output in the Java Astion Launcher log. If it is missing from there, I would like to see the java code that execute the script.
Regards,
Mohammad
________________________________
From: Derwin Chiang <de...@yahoo-inc.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
Sent: Monday, December 5, 2011 7:47 PM
Subject: RE: Is it possible to run multiple grid jobs through a script
Thanks Alejandro
I know this idea is not a good practice.
But why we need this is just because that these implementation were came from other team,
And we don't have enough time to figure out everything in these implementation for rewriting the workflow in oozie.
So we wonder know can we simply migration it to oozie with little effort.
But if it's not supported by oozie, hmmm, OK, at least we know we would have a long way to go
Thanks.
Derwin
-----Original Message-----
From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
Sent: Tuesday, December 06, 2011 1:38 AM
To: oozie-users@incubator.apache.org
Subject: Re: Is it possible to run multiple grid jobs through a script
Derwin,
Doing as you describe it means it will be quite difficult for you track
where your jobs are and their status.
I'd recommend wiring all your hadoop/pig jobs in workflow and then using a
coordinator to trigger them.
Regards.
Alejandro
On Mon, Dec 5, 2011 at 3:06 AM, Derwin Chiang <de...@yahoo-inc.com> wrote:
> Hi, All
>
> I am doing migration for running cron job on grid,
> Originally, we wrote one script and run the script every day
> automatically, there are some commands for triggering hadoop/pig jobs in
> this script.
> Since we can run script on oozie through some java action,
> Is it possible that we directly run this script through this way ?
> I have taken some try, it seems that it can run this script, but no
> result output.
> Does anyone have this experience ? Or do you know how to do this?
> Thanks
>
> Derwin
>
RE: Is it possible to run multiple grid jobs through a script
Posted by Derwin Chiang <de...@yahoo-inc.com>.
Thanks Alejandro
I know this idea is not a good practice.
But why we need this is just because that these implementation were came from other team,
And we don't have enough time to figure out everything in these implementation for rewriting the workflow in oozie.
So we wonder know can we simply migration it to oozie with little effort.
But if it's not supported by oozie, hmmm, OK, at least we know we would have a long way to go
Thanks.
Derwin
-----Original Message-----
From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
Sent: Tuesday, December 06, 2011 1:38 AM
To: oozie-users@incubator.apache.org
Subject: Re: Is it possible to run multiple grid jobs through a script
Derwin,
Doing as you describe it means it will be quite difficult for you track
where your jobs are and their status.
I'd recommend wiring all your hadoop/pig jobs in workflow and then using a
coordinator to trigger them.
Regards.
Alejandro
On Mon, Dec 5, 2011 at 3:06 AM, Derwin Chiang <de...@yahoo-inc.com> wrote:
> Hi, All
>
> I am doing migration for running cron job on grid,
> Originally, we wrote one script and run the script every day
> automatically, there are some commands for triggering hadoop/pig jobs in
> this script.
> Since we can run script on oozie through some java action,
> Is it possible that we directly run this script through this way ?
> I have taken some try, it seems that it can run this script, but no
> result output.
> Does anyone have this experience ? Or do you know how to do this?
> Thanks
>
> Derwin
>
Re: Is it possible to run multiple grid jobs through a script
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Derwin,
Doing as you describe it means it will be quite difficult for you track
where your jobs are and their status.
I'd recommend wiring all your hadoop/pig jobs in workflow and then using a
coordinator to trigger them.
Regards.
Alejandro
On Mon, Dec 5, 2011 at 3:06 AM, Derwin Chiang <de...@yahoo-inc.com> wrote:
> Hi, All
>
> I am doing migration for running cron job on grid,
> Originally, we wrote one script and run the script every day
> automatically, there are some commands for triggering hadoop/pig jobs in
> this script.
> Since we can run script on oozie through some java action,
> Is it possible that we directly run this script through this way ?
> I have taken some try, it seems that it can run this script, but no
> result output.
> Does anyone have this experience ? Or do you know how to do this?
> Thanks
>
> Derwin
>