You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Jakub Stransky <st...@gmail.com> on 2015/02/04 16:01:44 UTC
oozie 4.1.0 louncher fail with OozieLauncherInputFormat$EmptySplit
not found
Hello experienced oozie users,
we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We are
using simple authentication etc. When the job gets submitted oozie launcher
is failing with following:
2015-02-04 09:56:33,364 WARN [main]
org.apache.hadoop.mapred.YarnChild: Exception running child :
java.io.IOException: Split class
org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
found
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class
org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
... 7 more
2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
Runnning cleanup for the task
Found that this classes are contained in oozie share/lib uploaded to HDFS
to /user/${user.name}/share/lib according to oozie-site.xml configuration:
<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/${user.name}/share/lib</value>
</property>
where ${user.name} is user under which oozie server is running e.g. oozie.
For some reason those libs doesn't get propagated to
mapreduce.application.classpath and hence ClassNotFoundException.
Tried to change following property to true:
<property>
<name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
<value>true</value>
</property>
But with no difference. I am not sure what I am missing in installation
instruction here
<http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation>
Any hint would be appreciated.
Thx
Re: oozie 4.1.0 louncher fail with
OozieLauncherInputFormat$EmptySplit not found
Posted by Purshotam Shah <pu...@yahoo-inc.com.INVALID>.
You sharelib setup is correct and if you are still getting OozieLauncherInputFormat$EmptySplit not found, then i guess you might have old sharelib jars which doesn't contain OozieLauncherInputFormat.
Can you try with setting oozie.action.ship.launcher.jar = true in oozie-default.xml
<property> <name>oozie.action.ship.launcher.jar</name> <value>true</value> <description> Specifies whether launcher jar is shipped or not. </description> </property>
Puru.
From: Jakub Stransky <st...@gmail.com>
To: user@oozie.apache.org; Mohammad Islam <mi...@yahoo.com>
Sent: Wednesday, February 11, 2015 2:04 AM
Subject: Re: oozie 4.1.0 louncher fail with OozieLauncherInputFormat$EmptySplit not found
Hello experienced users,
anybody faced similar issue? I am stack on this for 3 day and still not
found what might be a problem. Since the installation of share libs is
correct as I see a corect folder with a timestamp but cannot figure it out
why this is not added to classpath even when I
specify oozie.use.system.libpath=true in job.properties. When tried
strace oozie
admin -shareliblist command I can see a failing child process but can not
nail it down.
Does anybody has some suggestion what to try or check?
On 6 February 2015 at 10:44, Jakub Stransky <st...@gmail.com> wrote:
> Hello,
>
> I tried a new installation procedure for shreLib installation after
> overcoming issues like: user oozie cannot impersonate user oozie etc. I got
> share/libs sucessfully installed to HDFS under
> /user/oozie/share/lib/lib_20141107094835 which I guess is fine. BUT the
> result of the command
> oozie admin -shareliblist
>
> returns null
>
> The commad for sharelib creation doesn't print any error to output nor
> oozie servers logs are free of any error. Can somebody help me to nail it
> down? Where oozie keep the information which share lib should be used?
>
> Thanks for helping me out
>
> Jakub
>
>
> On 4 February 2015 at 23:27, Jakub Stransky <st...@gmail.com> wrote:
>
>> No, I didn't I used instruction from apache oozie sites 4.1.0 which are
>> not different from the old one. WIll give a try!
>>
>> Thanks for pointing this out
>> Jakub
>>
>>
>> On 4 February 2015 at 22:55, Mohammad Islam <mi...@yahoo.com.invalid>
>> wrote:
>>
>>> Hi Jakub,Shared library structure was changed in 4.1.0.
>>> It is now in /user/oozie/shared/lib/lib_$timestamp.
>>> Did you follow the new instructions?Some discussion at:How-to: Use the
>>> ShareLib in Apache Oozie (CDH 5)
>>>
>>> | |
>>> | | | | | |
>>> | How-to: Use the ShareLib in Apache Oozie (CDH 5)drwxr-xr-x
>>> share/lib/lib_20140403151601/distcp -rw-r--r--
>>> share/lib/lib_20140403151601/distcp/hadoop-distcp-2.3.0-cdh5.0.0.jar |
>>> | |
>>> | View on blog.cloudera.com | Preview by Yahoo |
>>> | |
>>> | |
>>>
>>>
>>> Regards,Mohamamd
>>>
>>>
>>>
>>>
>>> On Wednesday, February 4, 2015 7:03 AM, Jakub Stransky <
>>> stransky.ja@gmail.com> wrote:
>>>
>>>
>>> Hello experienced oozie users,
>>>
>>> we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
>>> upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We are
>>> using simple authentication etc. When the job gets submitted oozie
>>> launcher
>>> is failing with following:
>>>
>>> 2015-02-04 09:56:33,364 WARN [main]
>>> org.apache.hadoop.mapred.YarnChild: Exception running child :
>>> java.io.IOException: Split class
>>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>>> found
>>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>>> Caused by: java.lang.ClassNotFoundException: Class
>>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>>> found
>>> at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
>>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
>>> ... 7 more
>>>
>>> 2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
>>> Runnning cleanup for the task
>>>
>>> Found that this classes are contained in oozie share/lib uploaded to HDFS
>>> to /user/${user.name}/share/lib according to oozie-site.xml
>>> configuration:
>>>
>>> <property>
>>> <name>oozie.service.WorkflowAppService.system.libpath</name>
>>> <value>/user/${user.name}/share/lib</value>
>>> </property>
>>>
>>> where ${user.name} is user under which oozie server is running e.g.
>>> oozie.
>>> For some reason those libs doesn't get propagated to
>>> mapreduce.application.classpath and hence ClassNotFoundException.
>>>
>>> Tried to change following property to true:
>>>
>>> <property>
>>> <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
>>> <value>true</value>
>>> </property>
>>>
>>> But with no difference. I am not sure what I am missing in installation
>>> instruction here
>>> <
>>> http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation
>>> >
>>>
>>> Any hint would be appreciated.
>>>
>>> Thx
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Jakub Stransky
>> cz.linkedin.com/in/jakubstransky
>>
>>
>
>
> --
> Jakub Stransky
> cz.linkedin.com/in/jakubstransky
>
>
--
Jakub Stransky
cz.linkedin.com/in/jakubstransky
Re: oozie 4.1.0 louncher fail with OozieLauncherInputFormat$EmptySplit
not found
Posted by Jakub Stransky <st...@gmail.com>.
I used oozie-default.xml from Hortonworks companion files and that was
missing
On 13 Feb 2015 20:35, "Mohammad Islam" <mi...@yahoo.com.invalid> wrote:
> "org.apache.oozie.service.ShareLibService," is already in the
> oozie-default.xml for oozie-4.1.0.
>
> What could be the reason? It comes in the oozie-core package.
> Regards,Mohammad
>
>
Re: oozie 4.1.0 louncher fail with
OozieLauncherInputFormat$EmptySplit not found
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
"org.apache.oozie.service.ShareLibService," is already in the oozie-default.xml for oozie-4.1.0.
What could be the reason? It comes in the oozie-core package.
Regards,Mohammad
Re: oozie 4.1.0 louncher fail with OozieLauncherInputFormat$EmptySplit
not found
Posted by Jakub Stransky <st...@gmail.com>.
Hi Mohammad,
already sorted out. Take a look
http://stackoverflow.com/questions/28324458/oozie-4-1-0-louncher-fail-with-oozielauncherinputformatemptysplit-not-found
Thanks for help
Jakub
On 13 February 2015 at 08:49, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> Not sure what could be the reason. I assume you are not using kerberos
> security.
> I can see /user/oozie/share/lib-new. Can you please remove this?Also , for
> the time being try to keep only the latest sub-dir /user/oozie/share/ (such
> as : /user/oozie/share/lib/lib_20150204063210/*). It could be even better
> to start the shared lib part from the beginning after removing all.
>
>
>
>
>
> On Thursday, February 12, 2015 12:12 AM, Jakub Stransky <
> stransky.ja@gmail.com> wrote:
>
>
> Hello,
> thanks for replies. I checked the version of share libs and according to
> version of pig it seemed to me that version is correct. We installed
> sharelib tar which is shipped with oozie 4.1.0. However i attached the lis
> of sharelibs listed by command $hadoop fs -lsr /user/oozie/share/
> When reading a source code I found that command oozie admin
> -shareliblist is calling a REST endpoint
> http://bd-prg-en1:11000/oozie/v2/admin/list_sharelib?lib=lib_20150204064003returns
> an {} which is empty response.
> For completeness I am attaching oozie configuration files as well.
> I am running out of ideas what might be wrong.
> Thanks for helping me outjakub
>
> On 11 February 2015 at 22:37, Mohammad Islam <mi...@yahoo.com.invalid>
> wrote:
>
> Can you please send the output snippets of $hadoop fs -lsr
> /user/oozie/share/
>
> On Wednesday, February 11, 2015 2:06 AM, Jakub Stransky <
> stransky.ja@gmail.com> wrote:
>
>
> Hello experienced users,
>
> anybody faced similar issue? I am stack on this for 3 day and still not
> found what might be a problem. Since the installation of share libs is
> correct as I see a corect folder with a timestamp but cannot figure it out
> why this is not added to classpath even when I
> specify oozie.use.system.libpath=true in job.properties. When tried
> strace oozie
> admin -shareliblist command I can see a failing child process but can not
> nail it down.
>
> Does anybody has some suggestion what to try or check?
>
> On 6 February 2015 at 10:44, Jakub Stransky <st...@gmail.com> wrote:
>
> > Hello,
> >
> > I tried a new installation procedure for shreLib installation after
> > overcoming issues like: user oozie cannot impersonate user oozie etc. I
> got
> > share/libs sucessfully installed to HDFS under
> > /user/oozie/share/lib/lib_20141107094835 which I guess is fine. BUT the
> > result of the command
> > oozie admin -shareliblist
> >
> > returns null
> >
> > The commad for sharelib creation doesn't print any error to output nor
> > oozie servers logs are free of any error. Can somebody help me to nail it
> > down? Where oozie keep the information which share lib should be used?
> >
> > Thanks for helping me out
> >
> > Jakub
> >
> >
> > On 4 February 2015 at 23:27, Jakub Stransky <st...@gmail.com>
> wrote:
> >
> >> No, I didn't I used instruction from apache oozie sites 4.1.0 which are
> >> not different from the old one. WIll give a try!
> >>
> >> Thanks for pointing this out
> >> Jakub
> >>
> >>
> >> On 4 February 2015 at 22:55, Mohammad Islam <mislam77@yahoo.com.invalid
> >
> >> wrote:
> >>
> >>> Hi Jakub,Shared library structure was changed in 4.1.0.
> >>> It is now in /user/oozie/shared/lib/lib_$timestamp.
> >>> Did you follow the new instructions?Some discussion at:How-to: Use the
> >>> ShareLib in Apache Oozie (CDH 5)
> >>>
> >>> | |
> >>> | | | | | |
> >>> | How-to: Use the ShareLib in Apache Oozie (CDH 5)drwxr-xr-x
> >>> share/lib/lib_20140403151601/distcp -rw-r--r--
> >>> share/lib/lib_20140403151601/distcp/hadoop-distcp-2.3.0-cdh5.0.0.jar |
> >>> | |
> >>> | View on blog.cloudera.com | Preview by Yahoo |
> >>> | |
> >>> | |
> >>>
> >>>
> >>> Regards,Mohamamd
> >>>
> >>>
> >>>
> >>>
> >>> On Wednesday, February 4, 2015 7:03 AM, Jakub Stransky <
> >>> stransky.ja@gmail.com> wrote:
> >>>
> >>>
> >>> Hello experienced oozie users,
> >>>
> >>> we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
> >>> upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We
> are
> >>> using simple authentication etc. When the job gets submitted oozie
> >>> launcher
> >>> is failing with following:
> >>>
> >>> 2015-02-04 09:56:33,364 WARN [main]
> >>> org.apache.hadoop.mapred.YarnChild: Exception running child :
> >>> java.io.IOException: Split class
> >>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
> >>> found
> >>> at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
> >>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
> >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> >>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> >>> at java.security.AccessController.doPrivileged(Native Method)
> >>> at javax.security.auth.Subject.doAs(Subject.java:415)
> >>> at
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> >>> Caused by: java.lang.ClassNotFoundException: Class
> >>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
> >>> found
> >>> at
> >>>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
> >>> at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
> >>> ... 7 more
> >>>
> >>> 2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
> >>> Runnning cleanup for the task
> >>>
> >>> Found that this classes are contained in oozie share/lib uploaded to
> HDFS
> >>> to /user/${user.name}/share/lib according to oozie-site.xml
> >>> configuration:
> >>>
> >>> <property>
> >>> <name>oozie.service.WorkflowAppService.system.libpath</name>
> >>> <value>/user/${user.name}/share/lib</value>
> >>> </property>
> >>>
> >>> where ${user.name} is user under which oozie server is running e.g.
> >>> oozie.
> >>> For some reason those libs doesn't get propagated to
> >>> mapreduce.application.classpath and hence ClassNotFoundException.
> >>>
> >>> Tried to change following property to true:
> >>>
> >>> <property>
> >>> <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
> >>> <value>true</value>
> >>> </property>
> >>>
> >>> But with no difference. I am not sure what I am missing in installation
> >>> instruction here
> >>> <
> >>>
> http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation
> >>> >
> >>>
> >>> Any hint would be appreciated.
> >>>
> >>> Thx
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Jakub Stransky
> >> cz.linkedin.com/in/jakubstransky
> >>
> >>
> >
> >
> > --
> > Jakub Stransky
> > cz.linkedin.com/in/jakubstransky
> >
> >
>
>
> --
> Jakub Stransky
> cz.linkedin.com/in/jakubstransky
>
>
>
>
>
>
> --
> Jakub Stranskycz.linkedin.com/in/jakubstransky
>
>
>
>
--
Jakub Stransky
cz.linkedin.com/in/jakubstransky
Re: oozie 4.1.0 louncher fail with
OozieLauncherInputFormat$EmptySplit not found
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Not sure what could be the reason. I assume you are not using kerberos security.
I can see /user/oozie/share/lib-new. Can you please remove this?Also , for the time being try to keep only the latest sub-dir /user/oozie/share/ (such as : /user/oozie/share/lib/lib_20150204063210/*). It could be even better to start the shared lib part from the beginning after removing all.
On Thursday, February 12, 2015 12:12 AM, Jakub Stransky <st...@gmail.com> wrote:
Hello,
thanks for replies. I checked the version of share libs and according to version of pig it seemed to me that version is correct. We installed sharelib tar which is shipped with oozie 4.1.0. However i attached the lis of sharelibs listed by command $hadoop fs -lsr /user/oozie/share/
When reading a source code I found that command oozie admin -shareliblist is calling a REST endpoint http://bd-prg-en1:11000/oozie/v2/admin/list_sharelib?lib=lib_20150204064003returns an {} which is empty response.
For completeness I am attaching oozie configuration files as well.
I am running out of ideas what might be wrong.
Thanks for helping me outjakub
On 11 February 2015 at 22:37, Mohammad Islam <mi...@yahoo.com.invalid> wrote:
Can you please send the output snippets of $hadoop fs -lsr /user/oozie/share/
On Wednesday, February 11, 2015 2:06 AM, Jakub Stransky <st...@gmail.com> wrote:
Hello experienced users,
anybody faced similar issue? I am stack on this for 3 day and still not
found what might be a problem. Since the installation of share libs is
correct as I see a corect folder with a timestamp but cannot figure it out
why this is not added to classpath even when I
specify oozie.use.system.libpath=true in job.properties. When tried
strace oozie
admin -shareliblist command I can see a failing child process but can not
nail it down.
Does anybody has some suggestion what to try or check?
On 6 February 2015 at 10:44, Jakub Stransky <st...@gmail.com> wrote:
> Hello,
>
> I tried a new installation procedure for shreLib installation after
> overcoming issues like: user oozie cannot impersonate user oozie etc. I got
> share/libs sucessfully installed to HDFS under
> /user/oozie/share/lib/lib_20141107094835 which I guess is fine. BUT the
> result of the command
> oozie admin -shareliblist
>
> returns null
>
> The commad for sharelib creation doesn't print any error to output nor
> oozie servers logs are free of any error. Can somebody help me to nail it
> down? Where oozie keep the information which share lib should be used?
>
> Thanks for helping me out
>
> Jakub
>
>
> On 4 February 2015 at 23:27, Jakub Stransky <st...@gmail.com> wrote:
>
>> No, I didn't I used instruction from apache oozie sites 4.1.0 which are
>> not different from the old one. WIll give a try!
>>
>> Thanks for pointing this out
>> Jakub
>>
>>
>> On 4 February 2015 at 22:55, Mohammad Islam <mi...@yahoo.com.invalid>
>> wrote:
>>
>>> Hi Jakub,Shared library structure was changed in 4.1.0.
>>> It is now in /user/oozie/shared/lib/lib_$timestamp.
>>> Did you follow the new instructions?Some discussion at:How-to: Use the
>>> ShareLib in Apache Oozie (CDH 5)
>>>
>>> | |
>>> | | | | | |
>>> | How-to: Use the ShareLib in Apache Oozie (CDH 5)drwxr-xr-x
>>> share/lib/lib_20140403151601/distcp -rw-r--r--
>>> share/lib/lib_20140403151601/distcp/hadoop-distcp-2.3.0-cdh5.0.0.jar |
>>> | |
>>> | View on blog.cloudera.com | Preview by Yahoo |
>>> | |
>>> | |
>>>
>>>
>>> Regards,Mohamamd
>>>
>>>
>>>
>>>
>>> On Wednesday, February 4, 2015 7:03 AM, Jakub Stransky <
>>> stransky.ja@gmail.com> wrote:
>>>
>>>
>>> Hello experienced oozie users,
>>>
>>> we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
>>> upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We are
>>> using simple authentication etc. When the job gets submitted oozie
>>> launcher
>>> is failing with following:
>>>
>>> 2015-02-04 09:56:33,364 WARN [main]
>>> org.apache.hadoop.mapred.YarnChild: Exception running child :
>>> java.io.IOException: Split class
>>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>>> found
>>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>>> Caused by: java.lang.ClassNotFoundException: Class
>>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>>> found
>>> at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
>>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
>>> ... 7 more
>>>
>>> 2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
>>> Runnning cleanup for the task
>>>
>>> Found that this classes are contained in oozie share/lib uploaded to HDFS
>>> to /user/${user.name}/share/lib according to oozie-site.xml
>>> configuration:
>>>
>>> <property>
>>> <name>oozie.service.WorkflowAppService.system.libpath</name>
>>> <value>/user/${user.name}/share/lib</value>
>>> </property>
>>>
>>> where ${user.name} is user under which oozie server is running e.g.
>>> oozie.
>>> For some reason those libs doesn't get propagated to
>>> mapreduce.application.classpath and hence ClassNotFoundException.
>>>
>>> Tried to change following property to true:
>>>
>>> <property>
>>> <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
>>> <value>true</value>
>>> </property>
>>>
>>> But with no difference. I am not sure what I am missing in installation
>>> instruction here
>>> <
>>> http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation
>>> >
>>>
>>> Any hint would be appreciated.
>>>
>>> Thx
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Jakub Stransky
>> cz.linkedin.com/in/jakubstransky
>>
>>
>
>
> --
> Jakub Stransky
> cz.linkedin.com/in/jakubstransky
>
>
--
Jakub Stransky
cz.linkedin.com/in/jakubstransky
--
Jakub Stranskycz.linkedin.com/in/jakubstransky
Re: oozie 4.1.0 louncher fail with OozieLauncherInputFormat$EmptySplit
not found
Posted by Jakub Stransky <st...@gmail.com>.
Hello,
thanks for replies. I checked the version of share libs and according to
version of pig it seemed to me that version is correct. We installed
sharelib tar which is shipped with oozie 4.1.0. However i attached the lis
of sharelibs listed by command $hadoop fs -lsr /user/oozie/share/
When reading a source code I found that command oozie admin -shareliblist
is calling a REST endpoint
http://bd-prg-en1:11000/oozie/v2/admin/list_sharelib?lib=lib_20150204064003
returns an {} which is empty response.
For completeness I am attaching oozie configuration files as well.
I am running out of ideas what might be wrong.
Thanks for helping me out
jakub
On 11 February 2015 at 22:37, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> Can you please send the output snippets of $hadoop fs -lsr
> /user/oozie/share/
>
> On Wednesday, February 11, 2015 2:06 AM, Jakub Stransky <
> stransky.ja@gmail.com> wrote:
>
>
> Hello experienced users,
>
> anybody faced similar issue? I am stack on this for 3 day and still not
> found what might be a problem. Since the installation of share libs is
> correct as I see a corect folder with a timestamp but cannot figure it out
> why this is not added to classpath even when I
> specify oozie.use.system.libpath=true in job.properties. When tried
> strace oozie
> admin -shareliblist command I can see a failing child process but can not
> nail it down.
>
> Does anybody has some suggestion what to try or check?
>
> On 6 February 2015 at 10:44, Jakub Stransky <st...@gmail.com> wrote:
>
> > Hello,
> >
> > I tried a new installation procedure for shreLib installation after
> > overcoming issues like: user oozie cannot impersonate user oozie etc. I
> got
> > share/libs sucessfully installed to HDFS under
> > /user/oozie/share/lib/lib_20141107094835 which I guess is fine. BUT the
> > result of the command
> > oozie admin -shareliblist
> >
> > returns null
> >
> > The commad for sharelib creation doesn't print any error to output nor
> > oozie servers logs are free of any error. Can somebody help me to nail it
> > down? Where oozie keep the information which share lib should be used?
> >
> > Thanks for helping me out
> >
> > Jakub
> >
> >
> > On 4 February 2015 at 23:27, Jakub Stransky <st...@gmail.com>
> wrote:
> >
> >> No, I didn't I used instruction from apache oozie sites 4.1.0 which are
> >> not different from the old one. WIll give a try!
> >>
> >> Thanks for pointing this out
> >> Jakub
> >>
> >>
> >> On 4 February 2015 at 22:55, Mohammad Islam <mislam77@yahoo.com.invalid
> >
> >> wrote:
> >>
> >>> Hi Jakub,Shared library structure was changed in 4.1.0.
> >>> It is now in /user/oozie/shared/lib/lib_$timestamp.
> >>> Did you follow the new instructions?Some discussion at:How-to: Use the
> >>> ShareLib in Apache Oozie (CDH 5)
> >>>
> >>> | |
> >>> | | | | | |
> >>> | How-to: Use the ShareLib in Apache Oozie (CDH 5)drwxr-xr-x
> >>> share/lib/lib_20140403151601/distcp -rw-r--r--
> >>> share/lib/lib_20140403151601/distcp/hadoop-distcp-2.3.0-cdh5.0.0.jar |
> >>> | |
> >>> | View on blog.cloudera.com | Preview by Yahoo |
> >>> | |
> >>> | |
> >>>
> >>>
> >>> Regards,Mohamamd
> >>>
> >>>
> >>>
> >>>
> >>> On Wednesday, February 4, 2015 7:03 AM, Jakub Stransky <
> >>> stransky.ja@gmail.com> wrote:
> >>>
> >>>
> >>> Hello experienced oozie users,
> >>>
> >>> we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
> >>> upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We
> are
> >>> using simple authentication etc. When the job gets submitted oozie
> >>> launcher
> >>> is failing with following:
> >>>
> >>> 2015-02-04 09:56:33,364 WARN [main]
> >>> org.apache.hadoop.mapred.YarnChild: Exception running child :
> >>> java.io.IOException: Split class
> >>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
> >>> found
> >>> at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
> >>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
> >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> >>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> >>> at java.security.AccessController.doPrivileged(Native Method)
> >>> at javax.security.auth.Subject.doAs(Subject.java:415)
> >>> at
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> >>> Caused by: java.lang.ClassNotFoundException: Class
> >>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
> >>> found
> >>> at
> >>>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
> >>> at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
> >>> ... 7 more
> >>>
> >>> 2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
> >>> Runnning cleanup for the task
> >>>
> >>> Found that this classes are contained in oozie share/lib uploaded to
> HDFS
> >>> to /user/${user.name}/share/lib according to oozie-site.xml
> >>> configuration:
> >>>
> >>> <property>
> >>> <name>oozie.service.WorkflowAppService.system.libpath</name>
> >>> <value>/user/${user.name}/share/lib</value>
> >>> </property>
> >>>
> >>> where ${user.name} is user under which oozie server is running e.g.
> >>> oozie.
> >>> For some reason those libs doesn't get propagated to
> >>> mapreduce.application.classpath and hence ClassNotFoundException.
> >>>
> >>> Tried to change following property to true:
> >>>
> >>> <property>
> >>> <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
> >>> <value>true</value>
> >>> </property>
> >>>
> >>> But with no difference. I am not sure what I am missing in installation
> >>> instruction here
> >>> <
> >>>
> http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation
> >>> >
> >>>
> >>> Any hint would be appreciated.
> >>>
> >>> Thx
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Jakub Stransky
> >> cz.linkedin.com/in/jakubstransky
> >>
> >>
> >
> >
> > --
> > Jakub Stransky
> > cz.linkedin.com/in/jakubstransky
> >
> >
>
>
> --
> Jakub Stransky
> cz.linkedin.com/in/jakubstransky
>
>
>
>
--
Jakub Stransky
cz.linkedin.com/in/jakubstransky
Re: oozie 4.1.0 louncher fail with
OozieLauncherInputFormat$EmptySplit not found
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Can you please send the output snippets of $hadoop fs -lsr /user/oozie/share/
On Wednesday, February 11, 2015 2:06 AM, Jakub Stransky <st...@gmail.com> wrote:
Hello experienced users,
anybody faced similar issue? I am stack on this for 3 day and still not
found what might be a problem. Since the installation of share libs is
correct as I see a corect folder with a timestamp but cannot figure it out
why this is not added to classpath even when I
specify oozie.use.system.libpath=true in job.properties. When tried
strace oozie
admin -shareliblist command I can see a failing child process but can not
nail it down.
Does anybody has some suggestion what to try or check?
On 6 February 2015 at 10:44, Jakub Stransky <st...@gmail.com> wrote:
> Hello,
>
> I tried a new installation procedure for shreLib installation after
> overcoming issues like: user oozie cannot impersonate user oozie etc. I got
> share/libs sucessfully installed to HDFS under
> /user/oozie/share/lib/lib_20141107094835 which I guess is fine. BUT the
> result of the command
> oozie admin -shareliblist
>
> returns null
>
> The commad for sharelib creation doesn't print any error to output nor
> oozie servers logs are free of any error. Can somebody help me to nail it
> down? Where oozie keep the information which share lib should be used?
>
> Thanks for helping me out
>
> Jakub
>
>
> On 4 February 2015 at 23:27, Jakub Stransky <st...@gmail.com> wrote:
>
>> No, I didn't I used instruction from apache oozie sites 4.1.0 which are
>> not different from the old one. WIll give a try!
>>
>> Thanks for pointing this out
>> Jakub
>>
>>
>> On 4 February 2015 at 22:55, Mohammad Islam <mi...@yahoo.com.invalid>
>> wrote:
>>
>>> Hi Jakub,Shared library structure was changed in 4.1.0.
>>> It is now in /user/oozie/shared/lib/lib_$timestamp.
>>> Did you follow the new instructions?Some discussion at:How-to: Use the
>>> ShareLib in Apache Oozie (CDH 5)
>>>
>>> | |
>>> | | | | | |
>>> | How-to: Use the ShareLib in Apache Oozie (CDH 5)drwxr-xr-x
>>> share/lib/lib_20140403151601/distcp -rw-r--r--
>>> share/lib/lib_20140403151601/distcp/hadoop-distcp-2.3.0-cdh5.0.0.jar |
>>> | |
>>> | View on blog.cloudera.com | Preview by Yahoo |
>>> | |
>>> | |
>>>
>>>
>>> Regards,Mohamamd
>>>
>>>
>>>
>>>
>>> On Wednesday, February 4, 2015 7:03 AM, Jakub Stransky <
>>> stransky.ja@gmail.com> wrote:
>>>
>>>
>>> Hello experienced oozie users,
>>>
>>> we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
>>> upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We are
>>> using simple authentication etc. When the job gets submitted oozie
>>> launcher
>>> is failing with following:
>>>
>>> 2015-02-04 09:56:33,364 WARN [main]
>>> org.apache.hadoop.mapred.YarnChild: Exception running child :
>>> java.io.IOException: Split class
>>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>>> found
>>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>>> Caused by: java.lang.ClassNotFoundException: Class
>>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>>> found
>>> at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
>>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
>>> ... 7 more
>>>
>>> 2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
>>> Runnning cleanup for the task
>>>
>>> Found that this classes are contained in oozie share/lib uploaded to HDFS
>>> to /user/${user.name}/share/lib according to oozie-site.xml
>>> configuration:
>>>
>>> <property>
>>> <name>oozie.service.WorkflowAppService.system.libpath</name>
>>> <value>/user/${user.name}/share/lib</value>
>>> </property>
>>>
>>> where ${user.name} is user under which oozie server is running e.g.
>>> oozie.
>>> For some reason those libs doesn't get propagated to
>>> mapreduce.application.classpath and hence ClassNotFoundException.
>>>
>>> Tried to change following property to true:
>>>
>>> <property>
>>> <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
>>> <value>true</value>
>>> </property>
>>>
>>> But with no difference. I am not sure what I am missing in installation
>>> instruction here
>>> <
>>> http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation
>>> >
>>>
>>> Any hint would be appreciated.
>>>
>>> Thx
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Jakub Stransky
>> cz.linkedin.com/in/jakubstransky
>>
>>
>
>
> --
> Jakub Stransky
> cz.linkedin.com/in/jakubstransky
>
>
--
Jakub Stransky
cz.linkedin.com/in/jakubstransky
Re: oozie 4.1.0 louncher fail with OozieLauncherInputFormat$EmptySplit
not found
Posted by Jakub Stransky <st...@gmail.com>.
Hello experienced users,
anybody faced similar issue? I am stack on this for 3 day and still not
found what might be a problem. Since the installation of share libs is
correct as I see a corect folder with a timestamp but cannot figure it out
why this is not added to classpath even when I
specify oozie.use.system.libpath=true in job.properties. When tried
strace oozie
admin -shareliblist command I can see a failing child process but can not
nail it down.
Does anybody has some suggestion what to try or check?
On 6 February 2015 at 10:44, Jakub Stransky <st...@gmail.com> wrote:
> Hello,
>
> I tried a new installation procedure for shreLib installation after
> overcoming issues like: user oozie cannot impersonate user oozie etc. I got
> share/libs sucessfully installed to HDFS under
> /user/oozie/share/lib/lib_20141107094835 which I guess is fine. BUT the
> result of the command
> oozie admin -shareliblist
>
> returns null
>
> The commad for sharelib creation doesn't print any error to output nor
> oozie servers logs are free of any error. Can somebody help me to nail it
> down? Where oozie keep the information which share lib should be used?
>
> Thanks for helping me out
>
> Jakub
>
>
> On 4 February 2015 at 23:27, Jakub Stransky <st...@gmail.com> wrote:
>
>> No, I didn't I used instruction from apache oozie sites 4.1.0 which are
>> not different from the old one. WIll give a try!
>>
>> Thanks for pointing this out
>> Jakub
>>
>>
>> On 4 February 2015 at 22:55, Mohammad Islam <mi...@yahoo.com.invalid>
>> wrote:
>>
>>> Hi Jakub,Shared library structure was changed in 4.1.0.
>>> It is now in /user/oozie/shared/lib/lib_$timestamp.
>>> Did you follow the new instructions?Some discussion at:How-to: Use the
>>> ShareLib in Apache Oozie (CDH 5)
>>>
>>> | |
>>> | | | | | |
>>> | How-to: Use the ShareLib in Apache Oozie (CDH 5)drwxr-xr-x
>>> share/lib/lib_20140403151601/distcp -rw-r--r--
>>> share/lib/lib_20140403151601/distcp/hadoop-distcp-2.3.0-cdh5.0.0.jar |
>>> | |
>>> | View on blog.cloudera.com | Preview by Yahoo |
>>> | |
>>> | |
>>>
>>>
>>> Regards,Mohamamd
>>>
>>>
>>>
>>>
>>> On Wednesday, February 4, 2015 7:03 AM, Jakub Stransky <
>>> stransky.ja@gmail.com> wrote:
>>>
>>>
>>> Hello experienced oozie users,
>>>
>>> we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
>>> upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We are
>>> using simple authentication etc. When the job gets submitted oozie
>>> launcher
>>> is failing with following:
>>>
>>> 2015-02-04 09:56:33,364 WARN [main]
>>> org.apache.hadoop.mapred.YarnChild: Exception running child :
>>> java.io.IOException: Split class
>>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>>> found
>>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>>> Caused by: java.lang.ClassNotFoundException: Class
>>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>>> found
>>> at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
>>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
>>> ... 7 more
>>>
>>> 2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
>>> Runnning cleanup for the task
>>>
>>> Found that this classes are contained in oozie share/lib uploaded to HDFS
>>> to /user/${user.name}/share/lib according to oozie-site.xml
>>> configuration:
>>>
>>> <property>
>>> <name>oozie.service.WorkflowAppService.system.libpath</name>
>>> <value>/user/${user.name}/share/lib</value>
>>> </property>
>>>
>>> where ${user.name} is user under which oozie server is running e.g.
>>> oozie.
>>> For some reason those libs doesn't get propagated to
>>> mapreduce.application.classpath and hence ClassNotFoundException.
>>>
>>> Tried to change following property to true:
>>>
>>> <property>
>>> <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
>>> <value>true</value>
>>> </property>
>>>
>>> But with no difference. I am not sure what I am missing in installation
>>> instruction here
>>> <
>>> http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation
>>> >
>>>
>>> Any hint would be appreciated.
>>>
>>> Thx
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Jakub Stransky
>> cz.linkedin.com/in/jakubstransky
>>
>>
>
>
> --
> Jakub Stransky
> cz.linkedin.com/in/jakubstransky
>
>
--
Jakub Stransky
cz.linkedin.com/in/jakubstransky
Re: oozie 4.1.0 louncher fail with OozieLauncherInputFormat$EmptySplit
not found
Posted by Jakub Stransky <st...@gmail.com>.
Hello,
I tried a new installation procedure for shreLib installation after
overcoming issues like: user oozie cannot impersonate user oozie etc. I got
share/libs sucessfully installed to HDFS under
/user/oozie/share/lib/lib_20141107094835 which I guess is fine. BUT the
result of the command
oozie admin -shareliblist
returns null
The commad for sharelib creation doesn't print any error to output nor
oozie servers logs are free of any error. Can somebody help me to nail it
down? Where oozie keep the information which share lib should be used?
Thanks for helping me out
Jakub
On 4 February 2015 at 23:27, Jakub Stransky <st...@gmail.com> wrote:
> No, I didn't I used instruction from apache oozie sites 4.1.0 which are
> not different from the old one. WIll give a try!
>
> Thanks for pointing this out
> Jakub
>
>
> On 4 February 2015 at 22:55, Mohammad Islam <mi...@yahoo.com.invalid>
> wrote:
>
>> Hi Jakub,Shared library structure was changed in 4.1.0.
>> It is now in /user/oozie/shared/lib/lib_$timestamp.
>> Did you follow the new instructions?Some discussion at:How-to: Use the
>> ShareLib in Apache Oozie (CDH 5)
>>
>> | |
>> | | | | | |
>> | How-to: Use the ShareLib in Apache Oozie (CDH 5)drwxr-xr-x
>> share/lib/lib_20140403151601/distcp -rw-r--r--
>> share/lib/lib_20140403151601/distcp/hadoop-distcp-2.3.0-cdh5.0.0.jar |
>> | |
>> | View on blog.cloudera.com | Preview by Yahoo |
>> | |
>> | |
>>
>>
>> Regards,Mohamamd
>>
>>
>>
>>
>> On Wednesday, February 4, 2015 7:03 AM, Jakub Stransky <
>> stransky.ja@gmail.com> wrote:
>>
>>
>> Hello experienced oozie users,
>>
>> we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
>> upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We are
>> using simple authentication etc. When the job gets submitted oozie
>> launcher
>> is failing with following:
>>
>> 2015-02-04 09:56:33,364 WARN [main]
>> org.apache.hadoop.mapred.YarnChild: Exception running child :
>> java.io.IOException: Split class
>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>> found
>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>> Caused by: java.lang.ClassNotFoundException: Class
>> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
>> found
>> at
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
>> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
>> ... 7 more
>>
>> 2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
>> Runnning cleanup for the task
>>
>> Found that this classes are contained in oozie share/lib uploaded to HDFS
>> to /user/${user.name}/share/lib according to oozie-site.xml
>> configuration:
>>
>> <property>
>> <name>oozie.service.WorkflowAppService.system.libpath</name>
>> <value>/user/${user.name}/share/lib</value>
>> </property>
>>
>> where ${user.name} is user under which oozie server is running e.g.
>> oozie.
>> For some reason those libs doesn't get propagated to
>> mapreduce.application.classpath and hence ClassNotFoundException.
>>
>> Tried to change following property to true:
>>
>> <property>
>> <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
>> <value>true</value>
>> </property>
>>
>> But with no difference. I am not sure what I am missing in installation
>> instruction here
>> <
>> http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation
>> >
>>
>> Any hint would be appreciated.
>>
>> Thx
>>
>>
>>
>>
>
>
>
> --
> Jakub Stransky
> cz.linkedin.com/in/jakubstransky
>
>
--
Jakub Stransky
cz.linkedin.com/in/jakubstransky
Re: oozie 4.1.0 louncher fail with OozieLauncherInputFormat$EmptySplit
not found
Posted by Jakub Stransky <st...@gmail.com>.
No, I didn't I used instruction from apache oozie sites 4.1.0 which are not
different from the old one. WIll give a try!
Thanks for pointing this out
Jakub
On 4 February 2015 at 22:55, Mohammad Islam <mi...@yahoo.com.invalid>
wrote:
> Hi Jakub,Shared library structure was changed in 4.1.0.
> It is now in /user/oozie/shared/lib/lib_$timestamp.
> Did you follow the new instructions?Some discussion at:How-to: Use the
> ShareLib in Apache Oozie (CDH 5)
>
> | |
> | | | | | |
> | How-to: Use the ShareLib in Apache Oozie (CDH 5)drwxr-xr-x
> share/lib/lib_20140403151601/distcp -rw-r--r--
> share/lib/lib_20140403151601/distcp/hadoop-distcp-2.3.0-cdh5.0.0.jar |
> | |
> | View on blog.cloudera.com | Preview by Yahoo |
> | |
> | |
>
>
> Regards,Mohamamd
>
>
>
>
> On Wednesday, February 4, 2015 7:03 AM, Jakub Stransky <
> stransky.ja@gmail.com> wrote:
>
>
> Hello experienced oozie users,
>
> we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
> upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We are
> using simple authentication etc. When the job gets submitted oozie launcher
> is failing with following:
>
> 2015-02-04 09:56:33,364 WARN [main]
> org.apache.hadoop.mapred.YarnChild: Exception running child :
> java.io.IOException: Split class
> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
> found
> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
> found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
> at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
> ... 7 more
>
> 2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
> Runnning cleanup for the task
>
> Found that this classes are contained in oozie share/lib uploaded to HDFS
> to /user/${user.name}/share/lib according to oozie-site.xml configuration:
>
> <property>
> <name>oozie.service.WorkflowAppService.system.libpath</name>
> <value>/user/${user.name}/share/lib</value>
> </property>
>
> where ${user.name} is user under which oozie server is running e.g. oozie.
> For some reason those libs doesn't get propagated to
> mapreduce.application.classpath and hence ClassNotFoundException.
>
> Tried to change following property to true:
>
> <property>
> <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
> <value>true</value>
> </property>
>
> But with no difference. I am not sure what I am missing in installation
> instruction here
> <
> http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation
> >
>
> Any hint would be appreciated.
>
> Thx
>
>
>
>
--
Jakub Stransky
cz.linkedin.com/in/jakubstransky
Re: oozie 4.1.0 louncher fail with
OozieLauncherInputFormat$EmptySplit not found
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Hi Jakub,Shared library structure was changed in 4.1.0.
It is now in /user/oozie/shared/lib/lib_$timestamp.
Did you follow the new instructions?Some discussion at:How-to: Use the ShareLib in Apache Oozie (CDH 5)
| |
| | | | | |
| How-to: Use the ShareLib in Apache Oozie (CDH 5)drwxr-xr-x share/lib/lib_20140403151601/distcp -rw-r--r-- share/lib/lib_20140403151601/distcp/hadoop-distcp-2.3.0-cdh5.0.0.jar |
| |
| View on blog.cloudera.com | Preview by Yahoo |
| |
| |
Regards,Mohamamd
On Wednesday, February 4, 2015 7:03 AM, Jakub Stransky <st...@gmail.com> wrote:
Hello experienced oozie users,
we have upgraded our infrastructure from hdp 2.0 to hdp 2.2 which means
upgrade from hadoop 2.2.0 to 2.6.0 and oozie from 4.0.0 to 4.1.0. We are
using simple authentication etc. When the job gets submitted oozie launcher
is failing with following:
2015-02-04 09:56:33,364 WARN [main]
org.apache.hadoop.mapred.YarnChild: Exception running child :
java.io.IOException: Split class
org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
found
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:363)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:423)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class
org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit not
found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1982)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:361)
... 7 more
2015-02-04 09:56:33,373 INFO [main] org.apache.hadoop.mapred.Task:
Runnning cleanup for the task
Found that this classes are contained in oozie share/lib uploaded to HDFS
to /user/${user.name}/share/lib according to oozie-site.xml configuration:
<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/${user.name}/share/lib</value>
</property>
where ${user.name} is user under which oozie server is running e.g. oozie.
For some reason those libs doesn't get propagated to
mapreduce.application.classpath and hence ClassNotFoundException.
Tried to change following property to true:
<property>
<name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
<value>true</value>
</property>
But with no difference. I am not sure what I am missing in installation
instruction here
<http://oozie.apache.org/docs/4.1.0/DG_QuickStart.html#Oozie_Share_Lib_Installation>
Any hint would be appreciated.
Thx