You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "Mendelson, Assaf" <As...@rsa.com> on 2017/05/10 12:40:00 UTC

[WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Hi all,
When running spark I get the following warning: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Now I know that in general it is possible to ignore this warning, however, it means that utilities that catch "WARN" in the log keep flagging this.
I saw many answers to handling this (e.g. http://stackoverflow.com/questions/30369380/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-docker, http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning, http://stackoverflow.com/questions/40015416/spark-unable-to-load-native-hadoop-library-for-your-platform), however, I am unable to solve this on my local machine.
Specifically, I can't find any such solution for windows (i.e. when running developer local builds) or on a centos 7 machine with no HDFS (basically it is a single node machine which uses spark standalone for testing).

Any help would be appreciated.

Thanks,
              Assaf.

RE: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Posted by "Mendelson, Assaf" <As...@rsa.com>.

Thanks for the response.
I will try with log4j.
That said, I am running in windows using winutil.exe and still getting the warning.

Thanks,
              Assaf.

From: Steve Loughran [mailto:stevel@hortonworks.com]
Sent: Tuesday, May 16, 2017 6:55 PM
To: Mendelson, Assaf
Cc: user@spark.apache.org
Subject: Re: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

On 10 May 2017, at 13:40, Mendelson, Assaf <as...@rsa.com>> wrote:

Hi all,
When running spark I get the following warning: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Now I know that in general it is possible to ignore this warning, however, it means that utilities that catch “WARN” in the log keep flagging this.
I saw many answers to handling this (e.g. http://stackoverflow.com/questions/30369380/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-docker, http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning,http://stackoverflow.com/questions/40015416/spark-unable-to-load-native-hadoop-library-for-your-platform), however, I am unable to solve this on my local machine.
Specifically, I can’t find any such solution for windows (i.e. when running developer local builds) or on a centos 7 machine with no HDFS (basically it is a single node machine which uses spark standalone for testing).

Log4J is your friend. I usually have (at least)

log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN

if you are working on Windows though, you do actually need the native libraries an winutils.exe on your path, or things won't work

Any help would be appreciated.

Thanks,
              Assaf.

Re: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Posted by Steve Loughran <st...@hortonworks.com>.

On 10 May 2017, at 13:40, Mendelson, Assaf <as...@rsa.com>> wrote:

Hi all,
When running spark I get the following warning: [WARN] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Now I know that in general it is possible to ignore this warning, however, it means that utilities that catch “WARN” in the log keep flagging this.
I saw many answers to handling this (e.g. http://stackoverflow.com/questions/30369380/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-docker, http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning,http://stackoverflow.com/questions/40015416/spark-unable-to-load-native-hadoop-library-for-your-platform), however, I am unable to solve this on my local machine.
Specifically, I can’t find any such solution for windows (i.e. when running developer local builds) or on a centos 7 machine with no HDFS (basically it is a single node machine which uses spark standalone for testing).


Log4J is your friend. I usually have (at least)

log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN

if you are working on Windows though, you do actually need the native libraries an winutils.exe on your path, or things won't work

Any help would be appreciated.

Thanks,
              Assaf.