You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by 안석찬 <ah...@gmail.com> on 2017/06/30 10:32:29 UTC
YARN test failure
Hi,
I implemented YARN node labeling on REEF and passed the local tests.(`mvn test` on reef home)
And now I’m trying to test my code on pseudo-cluster environment.
I set up a single-node in a pseudo-distributed mode and ran an example code on it. It works well up to here.
But it fails on reef yarn test(reef/bin/runyarntest.sh). It says repeatedly like below.
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
I have a hard time finding the reason why reef can’t find the application classpath.
Could you help me figure out why?
more error messages:
$scan ./runyarntests.sh
java -cp :/Users/scan/Workspace/reef/lang/java/reef-tests/target/reef-tests-0.16.0-SNAPSHOT-test-jar-with-dependencies.jar:/Users/scan/hadoop/etc/hadoop:/Users/scan/hadoop/etc/hadoop:/Users/scan/hadoop/etc/hadoop:/Users/scan/hadoop/share/hadoop/common/lib/*:/Users/scan/hadoop/share/hadoop/common/*:/Users/scan/hadoop/share/hadoop/hdfs:/Users/scan/hadoop/share/hadoop/hdfs/lib/*:/Users/scan/hadoop/share/hadoop/hdfs/*:/Users/scan/hadoop/share/hadoop/yarn/lib/*:/Users/scan/hadoop/share/hadoop/yarn/*:/Users/scan/hadoop/share/hadoop/mapreduce/lib/*:/Users/scan/hadoop/share/hadoop/mapreduce/*:/Users/scan/hadoop/contrib/capacity-scheduler/*.jar:/Users/scan/hadoop/share/hadoop/yarn/*:/Users/scan/hadoop/share/hadoop/yarn/lib/* org.junit.runner.JUnitCore org.apache.reef.tests.AllTestsSuite
JUnit version 4.11
Jun 30, 2017 7:28:02 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/scan/Workspace/reef/lang/java/reef-tests/target/reef-tests-0.16.0-SNAPSHOT-test-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/scan/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]
Jun 30, 2017 7:28:03 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
…
hadoop related part of my .zshrc:
export HADOOP_HOME=/Users/scan/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_INSTALL=$HADOOP_HOME
export REEF_HOME=/Users/scan/Workspace/dummy/reef
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
RE: YARN test failure
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com.INVALID>.
Check if the path for HADOOP_CONF_DIR is set correctly in the yarn machine where you run the test.
Julia
-----Original Message-----
From: 안석찬 [mailto:ahnsc93@gmail.com]
Sent: Friday, June 30, 2017 3:32 AM
To: dev@reef.apache.org
Subject: YARN test failure
Hi,
I implemented YARN node labeling on REEF and passed the local tests.(`mvn test` on reef home) And now I’m trying to test my code on pseudo-cluster environment.
I set up a single-node in a pseudo-distributed mode and ran an example code on it. It works well up to here.
But it fails on reef yarn test(reef/bin/runyarntest.sh). It says repeatedly like below.
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
I have a hard time finding the reason why reef can’t find the application classpath.
Could you help me figure out why?
more error messages:
$scan ./runyarntests.sh
java -cp :/Users/scan/Workspace/reef/lang/java/reef-tests/target/reef-tests-0.16.0-SNAPSHOT-test-jar-with-dependencies.jar:/Users/scan/hadoop/etc/hadoop:/Users/scan/hadoop/etc/hadoop:/Users/scan/hadoop/etc/hadoop:/Users/scan/hadoop/share/hadoop/common/lib/*:/Users/scan/hadoop/share/hadoop/common/*:/Users/scan/hadoop/share/hadoop/hdfs:/Users/scan/hadoop/share/hadoop/hdfs/lib/*:/Users/scan/hadoop/share/hadoop/hdfs/*:/Users/scan/hadoop/share/hadoop/yarn/lib/*:/Users/scan/hadoop/share/hadoop/yarn/*:/Users/scan/hadoop/share/hadoop/mapreduce/lib/*:/Users/scan/hadoop/share/hadoop/mapreduce/*:/Users/scan/hadoop/contrib/capacity-scheduler/*.jar:/Users/scan/hadoop/share/hadoop/yarn/*:/Users/scan/hadoop/share/hadoop/yarn/lib/* org.junit.runner.JUnitCore org.apache.reef.tests.AllTestsSuite
JUnit version 4.11
Jun 30, 2017 7:28:02 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/scan/Workspace/reef/lang/java/reef-tests/target/reef-tests-0.16.0-SNAPSHOT-test-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/scan/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.slf4j.org%2Fcodes.html%23multiple_bindings&data=02%7C01%7CQiuhe.Wang%40microsoft.com%7C9192745026db4d2872b808d4bfa355c6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636344155623769306&sdata=zHWeZxV4tBTkcO0XqBLn380xbh9SP8lp26YdW7QajgE%3D&reserved=0 for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]
Jun 30, 2017 7:28:03 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment
INFO: Running tests on YARN
.Jun 30, 2017 7:28:03 PM org.apache.reef.runtime.yarn.YarnClasspathProvider <init>
SEVERE: YarnConfiguration.YARN_APPLICATION_CLASSPATH is empty. This indicates a broken cluster configuration.
EJun 30, 2017 7:28:03 PM org.apache.reef.tests.TestEnvironmentFactory getNewTestEnvironment …
hadoop related part of my .zshrc:
export HADOOP_HOME=/Users/scan/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_INSTALL=$HADOOP_HOME
export REEF_HOME=/Users/scan/Workspace/dummy/reef
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop