You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by "Heller, Chris" <ch...@akamai.com> on 2014/02/21 21:27:25 UTC
Question about using system lib path jars
I¹m writing a custom mapper/reducer which internally uses a Hive serde for
output. I have the Oozie shared libs located in HDFS in
/user/${user.name}/oozie/share/lib, and have set
oozie.use.system.libpath=true in my job configuration.
Still my job is failing because of missing dependencies.
I looked at the job configuration and noticed that the only system library
added to the class path is oozie/json-simple-1.1.jar.
Does Oozie inspect what action nodes are in a workflow, and sub-set the
class path based on need? Since this is a map-reduce job, I do not have any
hive actions in the workflow.
-Chris
Re: Question about using system lib path jars
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
>What if I want multiple sharelibs? Can that be a comma separated value?
Yes.
On 2/21/14, 1:10 PM, "Heller, Chris" <ch...@akamai.com> wrote:
>Setting that property worked. Thanks.
>
>What if I want multiple sharelibs? Can that be a comma separated value?
>
>
>
>On 2/21/14, 4:03 PM, "Purshotam Shah" <pu...@yahoo-inc.com> wrote:
>
>>Does your sharelib have hive?
>>
>>You can use admin command to list all available sharelib.
>>
>>Command : $oozie admin -oozie http://localhost:11000/oozie -shareliblist
>>
>>
>>At each action you need to specify which sharelib you want to use. Since
>>you are running as map-reduce, map-reduce action should have
>>"oozie.action.sharelib.for.map-reduce=hive,² property.
>>
>><property>
>> <name>oozie.action.sharelib.for.map-reduce</name>
>> <value>hive<value>
>> </property>
>>
>>
>>Thanks
>>
>>From: "Heller, Chris" <ch...@akamai.com>>
>>Reply-To: <us...@oozie.apache.org>>
>>Date: Friday, February 21, 2014 at 12:27 PM
>>To: "user@oozie.apache.org<ma...@oozie.apache.org>"
>><us...@oozie.apache.org>>
>>Subject: Question about using system lib path jars
>>
>>I¹m writing a custom mapper/reducer which internally uses a Hive serde
>>for output. I have the Oozie shared libs located in HDFS in
>>/user/${user.name}/oozie/share/lib, and have set
>>oozie.use.system.libpath=true in my job configuration.
>>
>>Still my job is failing because of missing dependencies.
>>
>>I looked at the job configuration and noticed that the only system
>>library added to the class path is oozie/json-simple-1.1.jar.
>>
>>Does Oozie inspect what action nodes are in a workflow, and sub-set the
>>class path based on need? Since this is a map-reduce job, I do not have
>>any hive actions in the workflow.
>>
>>-Chris
Re: Question about using system lib path jars
Posted by "Heller, Chris" <ch...@akamai.com>.
Setting that property worked. Thanks.
What if I want multiple sharelibs? Can that be a comma separated value?
On 2/21/14, 4:03 PM, "Purshotam Shah" <pu...@yahoo-inc.com> wrote:
>Does your sharelib have hive?
>
>You can use admin command to list all available sharelib.
>
>Command : $oozie admin -oozie http://localhost:11000/oozie -shareliblist
>
>
>At each action you need to specify which sharelib you want to use. Since
>you are running as map-reduce, map-reduce action should have
>"oozie.action.sharelib.for.map-reduce=hive,² property.
>
><property>
> <name>oozie.action.sharelib.for.map-reduce</name>
> <value>hive<value>
> </property>
>
>
>Thanks
>
>From: "Heller, Chris" <ch...@akamai.com>>
>Reply-To: <us...@oozie.apache.org>>
>Date: Friday, February 21, 2014 at 12:27 PM
>To: "user@oozie.apache.org<ma...@oozie.apache.org>"
><us...@oozie.apache.org>>
>Subject: Question about using system lib path jars
>
>I¹m writing a custom mapper/reducer which internally uses a Hive serde
>for output. I have the Oozie shared libs located in HDFS in
>/user/${user.name}/oozie/share/lib, and have set
>oozie.use.system.libpath=true in my job configuration.
>
>Still my job is failing because of missing dependencies.
>
>I looked at the job configuration and noticed that the only system
>library added to the class path is oozie/json-simple-1.1.jar.
>
>Does Oozie inspect what action nodes are in a workflow, and sub-set the
>class path based on need? Since this is a map-reduce job, I do not have
>any hive actions in the workflow.
>
>-Chris
Re: Question about using system lib path jars
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
Does your sharelib have hive?
You can use admin command to list all available sharelib.
Command : $oozie admin -oozie http://localhost:11000/oozie -shareliblist
At each action you need to specify which sharelib you want to use. Since you are running as map-reduce, map-reduce action should have "oozie.action.sharelib.for.map-reduce=hive,” property.
<property>
<name>oozie.action.sharelib.for.map-reduce</name>
<value>hive<value>
</property>
Thanks
From: "Heller, Chris" <ch...@akamai.com>>
Reply-To: <us...@oozie.apache.org>>
Date: Friday, February 21, 2014 at 12:27 PM
To: "user@oozie.apache.org<ma...@oozie.apache.org>" <us...@oozie.apache.org>>
Subject: Question about using system lib path jars
I’m writing a custom mapper/reducer which internally uses a Hive serde for output. I have the Oozie shared libs located in HDFS in /user/${user.name}/oozie/share/lib, and have set oozie.use.system.libpath=true in my job configuration.
Still my job is failing because of missing dependencies.
I looked at the job configuration and noticed that the only system library added to the class path is oozie/json-simple-1.1.jar.
Does Oozie inspect what action nodes are in a workflow, and sub-set the class path based on need? Since this is a map-reduce job, I do not have any hive actions in the workflow.
-Chris