You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by "Felix.徐" <yg...@gmail.com> on 2013/03/12 04:40:40 UTC
Capture the output of a Hive query?
Hi, I have 2 questions:
1.How can I capture the output(a few lines) of a Hive query and pass it to
the next action(e.g. a shell script) for further processing? The
capture-output seems only supports a properties file.
2.Is it possible to manually trigger the same workflow with different
parameters ?
Re: Capture the output of a Hive query?
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Felix,
On #1, Hive actions don't produce output back to oozie. If you need the
next action to handle the output, you should point to the next action the
output directory of the hive action.
ON #2, definitely, you can add ${VAR} in your workflow configs, and provide
the the VAR=VALUE in the job.properties used for the job submission (
http://oozie.apache.org/docs/3.3.1/WorkflowFunctionalSpec.html#a4_Parameterization_of_Workflows
)
Thx
On Mon, Mar 11, 2013 at 8:40 PM, Felix.徐 <yg...@gmail.com> wrote:
> Hi, I have 2 questions:
> 1.How can I capture the output(a few lines) of a Hive query and pass it to
> the next action(e.g. a shell script) for further processing? The
> capture-output seems only supports a properties file.
>
> 2.Is it possible to manually trigger the same workflow with different
> parameters ?
>
--
Alejandro
Re: Capture the output of a Hive query?
Posted by "Felix.徐" <yg...@gmail.com>.
Seems I find the solution here :
http://blog.cloudera.com/blog/2012/12/how-to-use-the-sharelib-in-apache-oozie/
and
http://grokbase.com/p/cloudera/cdh-user/125h1188yv/oozie-cant-find-hive-action-xsd
2013/3/13 Felix.徐 <yg...@gmail.com>
> Ok thanks!
>
> If oozie runs every task as a Mapreduce, does it mean that I have to
> install Hive on every machine if I want to execute Hive action?
>
> Oozie failed with NoClassDefFoundError:
> org/apache/hadoop/hive/cli/CliDriver currently.
>
>
> 2013/3/12 Alejandro Abdelnur <tu...@cloudera.com>
>
>> Felix,
>>
>> On #1, you typically don't want to run user code (apps) in the oozie
>> server. Also, by using an MR job for the shell action we get automatic
>> load
>> distribution. If you need to run an action outside of the cluster, then
>> look at the SSH action.
>>
>> On #2, whatever you print in an action will be in the log of the launcher
>> MR job for the action in the only map task the job has. From the Oozie
>> WebUI you get can get to the JT job page, from there you must navigate to
>> the map task log.
>>
>> Thx
>>
>>
>> On Tue, Mar 12, 2013 at 12:45 AM, Felix.徐 <yg...@gmail.com> wrote:
>>
>> > I have 2 more questions:
>> > 1.Why is shell action performed as a mapreduce? Isn't it supposed to
>> run as
>> > a shell script on master node?
>> >
>> > 2.Is there any way to print some log within actions which can be shown
>> in
>> > the web console , current log seems meaningless to me ..
>> >
>> > Thanks!
>> >
>> >
>> > 2013/3/12 Felix.徐 <yg...@gmail.com>
>> >
>> > > Hi, I have 2 questions:
>> > > 1.How can I capture the output(a few lines) of a Hive query and pass
>> it
>> > to
>> > > the next action(e.g. a shell script) for further processing? The
>> > > capture-output seems only supports a properties file.
>> > >
>> > > 2.Is it possible to manually trigger the same workflow with different
>> > > parameters ?
>> > >
>> >
>>
>>
>>
>> --
>> Alejandro
>>
>
>
Re: Capture the output of a Hive query?
Posted by "Felix.徐" <yg...@gmail.com>.
Ok thanks!
If oozie runs every task as a Mapreduce, does it mean that I have to
install Hive on every machine if I want to execute Hive action?
Oozie failed with NoClassDefFoundError:
org/apache/hadoop/hive/cli/CliDriver currently.
2013/3/12 Alejandro Abdelnur <tu...@cloudera.com>
> Felix,
>
> On #1, you typically don't want to run user code (apps) in the oozie
> server. Also, by using an MR job for the shell action we get automatic load
> distribution. If you need to run an action outside of the cluster, then
> look at the SSH action.
>
> On #2, whatever you print in an action will be in the log of the launcher
> MR job for the action in the only map task the job has. From the Oozie
> WebUI you get can get to the JT job page, from there you must navigate to
> the map task log.
>
> Thx
>
>
> On Tue, Mar 12, 2013 at 12:45 AM, Felix.徐 <yg...@gmail.com> wrote:
>
> > I have 2 more questions:
> > 1.Why is shell action performed as a mapreduce? Isn't it supposed to run
> as
> > a shell script on master node?
> >
> > 2.Is there any way to print some log within actions which can be shown in
> > the web console , current log seems meaningless to me ..
> >
> > Thanks!
> >
> >
> > 2013/3/12 Felix.徐 <yg...@gmail.com>
> >
> > > Hi, I have 2 questions:
> > > 1.How can I capture the output(a few lines) of a Hive query and pass it
> > to
> > > the next action(e.g. a shell script) for further processing? The
> > > capture-output seems only supports a properties file.
> > >
> > > 2.Is it possible to manually trigger the same workflow with different
> > > parameters ?
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Capture the output of a Hive query?
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Felix,
On #1, you typically don't want to run user code (apps) in the oozie
server. Also, by using an MR job for the shell action we get automatic load
distribution. If you need to run an action outside of the cluster, then
look at the SSH action.
On #2, whatever you print in an action will be in the log of the launcher
MR job for the action in the only map task the job has. From the Oozie
WebUI you get can get to the JT job page, from there you must navigate to
the map task log.
Thx
On Tue, Mar 12, 2013 at 12:45 AM, Felix.徐 <yg...@gmail.com> wrote:
> I have 2 more questions:
> 1.Why is shell action performed as a mapreduce? Isn't it supposed to run as
> a shell script on master node?
>
> 2.Is there any way to print some log within actions which can be shown in
> the web console , current log seems meaningless to me ..
>
> Thanks!
>
>
> 2013/3/12 Felix.徐 <yg...@gmail.com>
>
> > Hi, I have 2 questions:
> > 1.How can I capture the output(a few lines) of a Hive query and pass it
> to
> > the next action(e.g. a shell script) for further processing? The
> > capture-output seems only supports a properties file.
> >
> > 2.Is it possible to manually trigger the same workflow with different
> > parameters ?
> >
>
--
Alejandro
Re: Capture the output of a Hive query?
Posted by "Felix.徐" <yg...@gmail.com>.
I have 2 more questions:
1.Why is shell action performed as a mapreduce? Isn't it supposed to run as
a shell script on master node?
2.Is there any way to print some log within actions which can be shown in
the web console , current log seems meaningless to me ..
Thanks!
2013/3/12 Felix.徐 <yg...@gmail.com>
> Hi, I have 2 questions:
> 1.How can I capture the output(a few lines) of a Hive query and pass it to
> the next action(e.g. a shell script) for further processing? The
> capture-output seems only supports a properties file.
>
> 2.Is it possible to manually trigger the same workflow with different
> parameters ?
>