You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by xeon Mailinglist <xe...@gmail.com> on 2014/02/13 06:27:32 UTC
Unable to load native-hadoop library for your platform
I am trying to run an example and I get the following error:
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/02/13 05:24:48 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
I tried to run execstack -c, but the problem stays the same. Any help?
HadoopMaster-nh:~# execstack -c
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/02/13 05:26:45 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
RE: Unable to load native-hadoop library for your platform
Posted by Steve Kallestad <st...@tabtonic.com>.
Funny, I was just trying to add something to the wiki addressing this.
These instructions are for 2.2, but I imagine that 2.0.5 is probably very
similar.
If the formatting doesn't come through for whatever reason, I posted the
same thing here:
http://answers.splunk.com/answers/118174/hunk-reports-an-error-with-apache-hadoop?page=1&focusedAnswerId=122311#122311
This isn't necessarily a big problem - Hadoop will function without native
libraries. You may find it easier to ignore the message or
disable/redirect JVM warnings.
You can disable the error message or redirect it to stderr, but that only
moves the error out of your way and doesn't deal with the root problem. The
root problem is that the hadoop distribution does not include native
libraries. They must be compiled from source.
You can build your own distribution that includes native libraries using
the following steps:
1) Install developer tools and dependencies:
1a) From repositories:
apt-get install gcc g++ make maven cmake zlib zlib1g-dev
for RedHat environments, you can probably use a similar yum line:
yum install gcc g++ make maven cmake zlib zlib-devel
There may be some other dependencies or slightly different package names
depending on what you already have installed and what OS you are running.
If so, some google-able errors will pop up during the rest of the process.
1b) Protocol Buffers From Source:
mkdir /tmp/protobuf
cd /tmp/protobuf
wget http:// protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar -xvzf ./protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure --prefix=/usr
make
sudo make install
cd java
mvn install
mvn package
sudo ldconfig
cd /tmp
rm -rf protobuf
2) download hadoop source:
mkdir /tmp/hadoop-build
cd /tmp/hadoop-build
wget http://
apache.petsads.us/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
tar -xvzf ./hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src
3) Edit the hadoop-auth pom file.
vi hadoop-common-project/hadoop-auth/pom.xml
add the following dependency:
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
You should see an already existing dependency that looks very similar if
you search for "org.mortbay.jetty", add this dependency above or below it.
3) Compile it:
export Platform=x64
cd /tmp/hadoop-build/hadoop-2.2.0-src
mvn clean install -DskipTests
cd hadoop-mapreduce-project
mvn package -Pdist,native -DskipTests=true -Dtar
cd /tmp/hadoop-build/hadoop-2.2.0-src
mvn package -Pdist,native -DskipTests=true -Dtar
4) Copy your natively compiled distribution somewhere to be saved:
cp
/tmp/hadoop-build/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0.tar.gz
/my/distribution/share/hadoop-2.2.0.tar.gz
5) Delete the build files (once you are satisfied that everything is
working properly):
cd /tmp
rm -rf hadoop-build
Now any fresh installations based on this build will include native 64 bit
libraries. You can set up a new instance of hadoop locally, or you can
simply overwrite the files in the $HADOOP-INSTALL/lib/native directory with
those in your hadoop-2.2.0.tar.gz file.
*From:* xeon Mailinglist [mailto:xeonmailinglist@gmail.com]
*Sent:* Wednesday, February 12, 2014 9:28 PM
*To:* user@hadoop.apache.org
*Subject:* Unable to load native-hadoop library for your platform
I am trying to run an example and I get the following error:
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/02/13 05:24:48 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
I tried to run execstack -c, but the problem stays the same. Any help?
HadoopMaster-nh:~# execstack -c
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/02/13 05:26:45 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
RE: Unable to load native-hadoop library for your platform
Posted by Steve Kallestad <st...@tabtonic.com>.
Funny, I was just trying to add something to the wiki addressing this.
These instructions are for 2.2, but I imagine that 2.0.5 is probably very
similar.
If the formatting doesn't come through for whatever reason, I posted the
same thing here:
http://answers.splunk.com/answers/118174/hunk-reports-an-error-with-apache-hadoop?page=1&focusedAnswerId=122311#122311
This isn't necessarily a big problem - Hadoop will function without native
libraries. You may find it easier to ignore the message or
disable/redirect JVM warnings.
You can disable the error message or redirect it to stderr, but that only
moves the error out of your way and doesn't deal with the root problem. The
root problem is that the hadoop distribution does not include native
libraries. They must be compiled from source.
You can build your own distribution that includes native libraries using
the following steps:
1) Install developer tools and dependencies:
1a) From repositories:
apt-get install gcc g++ make maven cmake zlib zlib1g-dev
for RedHat environments, you can probably use a similar yum line:
yum install gcc g++ make maven cmake zlib zlib-devel
There may be some other dependencies or slightly different package names
depending on what you already have installed and what OS you are running.
If so, some google-able errors will pop up during the rest of the process.
1b) Protocol Buffers From Source:
mkdir /tmp/protobuf
cd /tmp/protobuf
wget http:// protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar -xvzf ./protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure --prefix=/usr
make
sudo make install
cd java
mvn install
mvn package
sudo ldconfig
cd /tmp
rm -rf protobuf
2) download hadoop source:
mkdir /tmp/hadoop-build
cd /tmp/hadoop-build
wget http://
apache.petsads.us/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
tar -xvzf ./hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src
3) Edit the hadoop-auth pom file.
vi hadoop-common-project/hadoop-auth/pom.xml
add the following dependency:
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
You should see an already existing dependency that looks very similar if
you search for "org.mortbay.jetty", add this dependency above or below it.
3) Compile it:
export Platform=x64
cd /tmp/hadoop-build/hadoop-2.2.0-src
mvn clean install -DskipTests
cd hadoop-mapreduce-project
mvn package -Pdist,native -DskipTests=true -Dtar
cd /tmp/hadoop-build/hadoop-2.2.0-src
mvn package -Pdist,native -DskipTests=true -Dtar
4) Copy your natively compiled distribution somewhere to be saved:
cp
/tmp/hadoop-build/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0.tar.gz
/my/distribution/share/hadoop-2.2.0.tar.gz
5) Delete the build files (once you are satisfied that everything is
working properly):
cd /tmp
rm -rf hadoop-build
Now any fresh installations based on this build will include native 64 bit
libraries. You can set up a new instance of hadoop locally, or you can
simply overwrite the files in the $HADOOP-INSTALL/lib/native directory with
those in your hadoop-2.2.0.tar.gz file.
*From:* xeon Mailinglist [mailto:xeonmailinglist@gmail.com]
*Sent:* Wednesday, February 12, 2014 9:28 PM
*To:* user@hadoop.apache.org
*Subject:* Unable to load native-hadoop library for your platform
I am trying to run an example and I get the following error:
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/02/13 05:24:48 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
I tried to run execstack -c, but the problem stays the same. Any help?
HadoopMaster-nh:~# execstack -c
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/02/13 05:26:45 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
RE: Unable to load native-hadoop library for your platform
Posted by Steve Kallestad <st...@tabtonic.com>.
Funny, I was just trying to add something to the wiki addressing this.
These instructions are for 2.2, but I imagine that 2.0.5 is probably very
similar.
If the formatting doesn't come through for whatever reason, I posted the
same thing here:
http://answers.splunk.com/answers/118174/hunk-reports-an-error-with-apache-hadoop?page=1&focusedAnswerId=122311#122311
This isn't necessarily a big problem - Hadoop will function without native
libraries. You may find it easier to ignore the message or
disable/redirect JVM warnings.
You can disable the error message or redirect it to stderr, but that only
moves the error out of your way and doesn't deal with the root problem. The
root problem is that the hadoop distribution does not include native
libraries. They must be compiled from source.
You can build your own distribution that includes native libraries using
the following steps:
1) Install developer tools and dependencies:
1a) From repositories:
apt-get install gcc g++ make maven cmake zlib zlib1g-dev
for RedHat environments, you can probably use a similar yum line:
yum install gcc g++ make maven cmake zlib zlib-devel
There may be some other dependencies or slightly different package names
depending on what you already have installed and what OS you are running.
If so, some google-able errors will pop up during the rest of the process.
1b) Protocol Buffers From Source:
mkdir /tmp/protobuf
cd /tmp/protobuf
wget http:// protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar -xvzf ./protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure --prefix=/usr
make
sudo make install
cd java
mvn install
mvn package
sudo ldconfig
cd /tmp
rm -rf protobuf
2) download hadoop source:
mkdir /tmp/hadoop-build
cd /tmp/hadoop-build
wget http://
apache.petsads.us/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
tar -xvzf ./hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src
3) Edit the hadoop-auth pom file.
vi hadoop-common-project/hadoop-auth/pom.xml
add the following dependency:
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
You should see an already existing dependency that looks very similar if
you search for "org.mortbay.jetty", add this dependency above or below it.
3) Compile it:
export Platform=x64
cd /tmp/hadoop-build/hadoop-2.2.0-src
mvn clean install -DskipTests
cd hadoop-mapreduce-project
mvn package -Pdist,native -DskipTests=true -Dtar
cd /tmp/hadoop-build/hadoop-2.2.0-src
mvn package -Pdist,native -DskipTests=true -Dtar
4) Copy your natively compiled distribution somewhere to be saved:
cp
/tmp/hadoop-build/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0.tar.gz
/my/distribution/share/hadoop-2.2.0.tar.gz
5) Delete the build files (once you are satisfied that everything is
working properly):
cd /tmp
rm -rf hadoop-build
Now any fresh installations based on this build will include native 64 bit
libraries. You can set up a new instance of hadoop locally, or you can
simply overwrite the files in the $HADOOP-INSTALL/lib/native directory with
those in your hadoop-2.2.0.tar.gz file.
*From:* xeon Mailinglist [mailto:xeonmailinglist@gmail.com]
*Sent:* Wednesday, February 12, 2014 9:28 PM
*To:* user@hadoop.apache.org
*Subject:* Unable to load native-hadoop library for your platform
I am trying to run an example and I get the following error:
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/02/13 05:24:48 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
I tried to run execstack -c, but the problem stays the same. Any help?
HadoopMaster-nh:~# execstack -c
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/02/13 05:26:45 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
RE: Unable to load native-hadoop library for your platform
Posted by Steve Kallestad <st...@tabtonic.com>.
Funny, I was just trying to add something to the wiki addressing this.
These instructions are for 2.2, but I imagine that 2.0.5 is probably very
similar.
If the formatting doesn't come through for whatever reason, I posted the
same thing here:
http://answers.splunk.com/answers/118174/hunk-reports-an-error-with-apache-hadoop?page=1&focusedAnswerId=122311#122311
This isn't necessarily a big problem - Hadoop will function without native
libraries. You may find it easier to ignore the message or
disable/redirect JVM warnings.
You can disable the error message or redirect it to stderr, but that only
moves the error out of your way and doesn't deal with the root problem. The
root problem is that the hadoop distribution does not include native
libraries. They must be compiled from source.
You can build your own distribution that includes native libraries using
the following steps:
1) Install developer tools and dependencies:
1a) From repositories:
apt-get install gcc g++ make maven cmake zlib zlib1g-dev
for RedHat environments, you can probably use a similar yum line:
yum install gcc g++ make maven cmake zlib zlib-devel
There may be some other dependencies or slightly different package names
depending on what you already have installed and what OS you are running.
If so, some google-able errors will pop up during the rest of the process.
1b) Protocol Buffers From Source:
mkdir /tmp/protobuf
cd /tmp/protobuf
wget http:// protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar -xvzf ./protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure --prefix=/usr
make
sudo make install
cd java
mvn install
mvn package
sudo ldconfig
cd /tmp
rm -rf protobuf
2) download hadoop source:
mkdir /tmp/hadoop-build
cd /tmp/hadoop-build
wget http://
apache.petsads.us/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
tar -xvzf ./hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src
3) Edit the hadoop-auth pom file.
vi hadoop-common-project/hadoop-auth/pom.xml
add the following dependency:
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
You should see an already existing dependency that looks very similar if
you search for "org.mortbay.jetty", add this dependency above or below it.
3) Compile it:
export Platform=x64
cd /tmp/hadoop-build/hadoop-2.2.0-src
mvn clean install -DskipTests
cd hadoop-mapreduce-project
mvn package -Pdist,native -DskipTests=true -Dtar
cd /tmp/hadoop-build/hadoop-2.2.0-src
mvn package -Pdist,native -DskipTests=true -Dtar
4) Copy your natively compiled distribution somewhere to be saved:
cp
/tmp/hadoop-build/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0.tar.gz
/my/distribution/share/hadoop-2.2.0.tar.gz
5) Delete the build files (once you are satisfied that everything is
working properly):
cd /tmp
rm -rf hadoop-build
Now any fresh installations based on this build will include native 64 bit
libraries. You can set up a new instance of hadoop locally, or you can
simply overwrite the files in the $HADOOP-INSTALL/lib/native directory with
those in your hadoop-2.2.0.tar.gz file.
*From:* xeon Mailinglist [mailto:xeonmailinglist@gmail.com]
*Sent:* Wednesday, February 12, 2014 9:28 PM
*To:* user@hadoop.apache.org
*Subject:* Unable to load native-hadoop library for your platform
I am trying to run an example and I get the following error:
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/02/13 05:24:48 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
I tried to run execstack -c, but the problem stays the same. Any help?
HadoopMaster-nh:~# execstack -c
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c
<libfile>', or link it with '-z noexecstack'.
14/02/13 05:26:45 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable