You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sophia <sl...@163.com> on 2014/05/07 03:25:09 UTC
Unable to load native-hadoop library problem
Hi,everyone,
[root@CHBM220 spark-0.9.1]#
SPARK_JAR=.assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
./bin/spark-class org.apache.spark.deploy.yarn.Client --jar
examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar --class
org.apache.spark.examples.SparkPi --args yarn-standalone --num-workers 3
--master-memory 2g --worker-memory 2g --worker-cores 1
14/05/07 09:05:14 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/05/07 09:05:14 INFO RMProxy: Connecting to ResourceManager at
CHBM220/192.168.10.220:8032
Then it stopped,my hadoop_conf_dir has been configued well,what should I do
to?
Wish you happy everyday.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-load-native-hadoop-library-problem-tp5469.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Unable to load native-hadoop library problem
Posted by Andrew Or <an...@databricks.com>.
This seems unrelated to not being able to load native-hadoop library. Is it
failing to connect to ResourceManager? Have you verified that there is an
RM process listening on port 8032 at the specified IP?
On Tue, May 6, 2014 at 6:25 PM, Sophia <sl...@163.com> wrote:
> Hi,everyone,
> [root@CHBM220 spark-0.9.1]#
>
> SPARK_JAR=.assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
> ./bin/spark-class org.apache.spark.deploy.yarn.Client --jar
> examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar --class
> org.apache.spark.examples.SparkPi --args yarn-standalone --num-workers 3
> --master-memory 2g --worker-memory 2g --worker-cores 1
> 14/05/07 09:05:14 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/05/07 09:05:14 INFO RMProxy: Connecting to ResourceManager at
> CHBM220/192.168.10.220:8032
> Then it stopped,my hadoop_conf_dir has been configued well,what should I do
> to?
> Wish you happy everyday.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-load-native-hadoop-library-problem-tp5469.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: Unable to load native-hadoop library problem
Posted by Shivani Rao <ra...@gmail.com>.
Hello Sophia
You are only providing the Spark jar here (nevertheless, a spark jar that
contains hadoop libraries in it, but that is not sufficient). Where is your
hadoop installed? (Most probably: /usr/lib/hadoop/*)
So you need to add that to your class path (by using -cp) I guess. Let me
know if that works
shivani
On Tue, May 6, 2014 at 6:25 PM, Sophia <sl...@163.com> wrote:
> Hi,everyone,
> [root@CHBM220 spark-0.9.1]#
>
> SPARK_JAR=.assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
> ./bin/spark-class org.apache.spark.deploy.yarn.Client --jar
> examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar --class
> org.apache.spark.examples.SparkPi --args yarn-standalone --num-workers 3
> --master-memory 2g --worker-memory 2g --worker-cores 1
> 14/05/07 09:05:14 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/05/07 09:05:14 INFO RMProxy: Connecting to ResourceManager at
> CHBM220/192.168.10.220:8032
> Then it stopped,my hadoop_conf_dir has been configued well,what should I do
> to?
> Wish you happy everyday.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-load-native-hadoop-library-problem-tp5469.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
--
Software Engineer
Analytics Engineering Team@ Box
Mountain View, CA