You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Attila Sasvari (JIRA)" <ji...@apache.org> on 2017/06/15 10:40:02 UTC
[jira] [Updated] (OOZIE-2944) Shell action example does not work
with Oozie on Yarn on hadoop 2.6
[ https://issues.apache.org/jira/browse/OOZIE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Attila Sasvari updated OOZIE-2944:
----------------------------------
Attachment: OOZIE-2944-01.patch
Tests:
- {{mvn test -DtargetJavaVersion=1.8 -DjavaVersion=1.8 -Dtest=TestLauncher,TestWorkflowActionKillXCommand,TestHadoopAccessorService -Dhadoop.version-2.6.0}} passed
- shell example succeeded
{code}
application_1497357254962_0071 asasvari oozie:launcher:T=shell:W=shell-wf:A=shell-node:ID=0000002-170615122337884-oozie-asas-W Oozie Launcher default Thu, 15 Jun 2017 10:26:30 GMT Thu, 15 Jun 2017 10:26:32 GMT FINISHED SUCCEEDED
{code}
> Shell action example does not work with Oozie on Yarn on hadoop 2.6
> -------------------------------------------------------------------
>
> Key: OOZIE-2944
> URL: https://issues.apache.org/jira/browse/OOZIE-2944
> Project: Oozie
> Issue Type: Bug
> Reporter: Attila Sasvari
> Assignee: Attila Sasvari
> Attachments: OOZIE-2944-01.patch
>
>
> I tried to execute example shell action with a pseudo hadoop 2.6.0. The workflow failed with the exception below :
> {code}
> $ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId application_1497357254962_0005
> 17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
> 17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Container: container_1497357254962_0005_01_000001 on 172.30.64.221_55437
> ==========================================================================
> LogType:stderr
> Log Upload Time:13-Jun-2017 15:58:45
> LogLength:1397
> Log Contents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/Mapper
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540)
> at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170)
> at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 15 more
> LogType:stdout
> Log Upload Time:13-Jun-2017 15:58:45
> LogLength:2152
> Log Contents:
> Submitter user is: asasvari
> log4j: Trying to find [container-log4j.properties] using context classloader sun.misc.Launcher$AppClassLoader@6bc7c054.
> log4j: Using URL [jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties] for automatic log4j configuration.
> log4j: Reading configuration from URL jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties
> log4j: Hierarchy threshold set to [ALL].
> log4j: Parsing for [root] with value=[INFO,CLA, EventCounter].
> log4j: Level token is [INFO].
> log4j: Category root set to INFO
> log4j: Parsing appender named "CLA".
> log4j: Parsing layout options for "CLA".
> log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n].
> log4j: End of parsing for "CLA".
> log4j: Setting property [containerLogDir] to [/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_000001].
> log4j: Setting property [totalLogFileSize] to [0].
> log4j: setFile called: /Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_000001/syslog, true
> log4j: setFile ended
> log4j: Parsed "CLA" options.
> log4j: Parsing appender named "EventCounter".
> log4j: Parsed "EventCounter" options.
> log4j: Finished configuring.
> Using login user for UGI
> Launcher AM configuration loaded
> Starting the execution of prepare actions
> Completed the execution of prepare actions successfully
> Failing Oozie Launcher, null
> Oozie Launcher, uploading action data to HDFS sequence file: hdfs://localhost:9000/user/asasvari/oozie-asas/0000002-170613150915408-oozie-asas-W/shell-node--shell/action-data.seq
> Stopping AM
> Callback notification attempts left 0
> Callback notification trying http://0.0.0.0:11000/oozie/callback?id=0000002-170613150915408-oozie-asas-W@shell-node&status=FAILED
> Callback notification to http://0.0.0.0:11000/oozie/callback?id=0000002-170613150915408-oozie-asas-W@shell-node&status=FAILED succeeded
> Callback notification succeeded
> LogType:syslog
> Log Upload Time:13-Jun-2017 15:58:45
> LogLength:1215
> Log Contents:
> 2017-06-13 15:58:43,105 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2017-06-13 15:58:43,271 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
> 2017-06-13 15:58:43,844 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
> 2017-06-13 15:58:43,972 INFO [main] org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
> 2017-06-13 15:58:44,078 INFO [AMRM Callback Handler Thread] org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: Interrupted while waiting for queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
> {code}
> I believe this issue is related to [OOZIE-2918|https://issues.apache.org/jira/browse/OOZIE-2918].
> {{ShellActionExecutor}} extends {{JavaActionExecutor}} whose {{needToAddMapReduceToClassPath()}} returns {{false}}. {{LauncherMapper.getMainArguments()}} is called from {{LauncherAM}} via its {{run()}} method. {{LauncherMapper}} extends {{org.apache.hadoop.mapred.Mapper}} that is not on the classpath.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)