You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Yuqi Gu (JIRA)" <ji...@apache.org> on 2019/08/05 08:27:00 UTC
[jira] [Comment Edited] (HADOOP-9320) Hadoop native build failure
on ARM hard-float
[ https://issues.apache.org/jira/browse/HADOOP-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899888#comment-16899888 ]
Yuqi Gu edited comment on HADOOP-9320 at 8/5/19 8:26 AM:
---------------------------------------------------------
We build the hadoop-2.8 native in our AArch64 platform:
Environment:
{code:java}
builder@yq-hadoop:~/hadoop$ arch
aarch64
builder@yq-hadoop:~/hadoop$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
{code}
# Install protobuf 2.5.0 for Arm64.
# Set JAVA_HOME:
{code:java}
export JAVA_HOME=`readlink -f /usr/bin/java | sed "s:jre/bin/java::"`
{code}
# Install openssl: libssl1.0-dev (Replace libssl1.0-dev with libssl-dev)
# Set BUNDLE_SNAPPY:
{code:java}
[ -f /usr/lib/libsnappy.so ] && BUNDLE_SNAPPY="-Dsnappy.prefix=x -Dbundle.snappy=true -Dsnappy.lib=/usr/lib"
[ -f /usr/lib64/libsnappy.so ] && BUNDLE_SNAPPY="-Dsnappy.prefix=x -Dbundle.snappy=true -Dsnappy.lib=/usr/lib64"
[ -f /usr/lib/aarch64-linux-gnu/libsnappy.so ] && BUNDLE_SNAPPY="-Dsnappy.prefix=x -Dbundle.snappy=true -Dsnappy.lib=/usr/lib/aarch64-linux-gnu"
{code}
Native build:
{code:java}
mvn clean install package $BUNDLE_SNAPPY -DskipTests -DskipITs -Pdist,native -Dtar -Dtar -Dmaven.javadoc.skip=true
{code}
We could successfully navie build Hadoop on 2.8:
{code:java}
[INFO] Executed tasks
[INFO]
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Skipping javadoc generation
[INFO]
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ hadoop-dist ---
[INFO]
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ hadoop-dist ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-dist ---
[INFO]
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-dist ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 2.8.5:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.334 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 0.849 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.211 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.571 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.250 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.230 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 2.473 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 4.845 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.866 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.068 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:13 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 1.560 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 22.003 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.087 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:14 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 36.665 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 26.192 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 29.175 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 21.165 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 1.609 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.080 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.079 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 7.357 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:00 min]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.078 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 2.653 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 16.461 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 1.155 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 17.939 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 10.252 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 1.800 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 1.973 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 1.713 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.534 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.074 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.320 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 0.794 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.076 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 1.928 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 10.448 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.504 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 5.500 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 3.511 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 1.239 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 3.884 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 2.529 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 9.822 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 0.830 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 1.477 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 6.919 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 4.892 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 9.796 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 0.544 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 0.757 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 1.383 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 1.177 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 0.561 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 0.254 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 0.752 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 18.946 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 1.297 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 16.051 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.903 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 8.313 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.995 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 2.028 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 13.643 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 8.190 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.070 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 31.118 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:12 min
[INFO] Finished at: 2019-08-05T08:14:15Z
[INFO] ------------------------------------------------------------------------
builder@yq-hadoop:~/hadoop$ arch
aarch64
builder@yq-hadoop:~/hadoop$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
{code}
was (Author: yqgu):
We build the hadoop native in our AArch64 platform:
Environment:
{code:java}
builder@yq-hadoop:~/hadoop$ arch
aarch64
builder@yq-hadoop:~/hadoop$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
{code}
# Install protobuf 2.5.0 for Arm64.
# Set JAVA_HOME:
{code:java}
export JAVA_HOME=`readlink -f /usr/bin/java | sed "s:jre/bin/java::"`
{code}
# Install openssl: libssl1.0-dev (Replace libssl1.0-dev with libssl-dev)
# Set BUNDLE_SNAPPY:
{code:java}
[ -f /usr/lib/libsnappy.so ] && BUNDLE_SNAPPY="-Dsnappy.prefix=x -Dbundle.snappy=true -Dsnappy.lib=/usr/lib"
[ -f /usr/lib64/libsnappy.so ] && BUNDLE_SNAPPY="-Dsnappy.prefix=x -Dbundle.snappy=true -Dsnappy.lib=/usr/lib64"
[ -f /usr/lib/aarch64-linux-gnu/libsnappy.so ] && BUNDLE_SNAPPY="-Dsnappy.prefix=x -Dbundle.snappy=true -Dsnappy.lib=/usr/lib/aarch64-linux-gnu"
{code}
Native build:
{code:java}
mvn clean install package $BUNDLE_SNAPPY -DskipTests -DskipITs -Pdist,native -Dtar -Dtar -Dmaven.javadoc.skip=true
{code}
We could successfully navie build Hadoop on 2.8:
{code:java}
[INFO] Executed tasks
[INFO]
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Skipping javadoc generation
[INFO]
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ hadoop-dist ---
[INFO]
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ hadoop-dist ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-dist ---
[INFO]
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-dist ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 2.8.5:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.334 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 0.849 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.211 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.571 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.250 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.230 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 2.473 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 4.845 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.866 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.068 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:13 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 1.560 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 22.003 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.087 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:14 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 36.665 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 26.192 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 29.175 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 21.165 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 1.609 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.080 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.079 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 7.357 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:00 min]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.078 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 2.653 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 16.461 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 1.155 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 17.939 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 10.252 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 1.800 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 1.973 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 1.713 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.534 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.074 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.320 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 0.794 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.076 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 1.928 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 10.448 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.504 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 5.500 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 3.511 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 1.239 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 3.884 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 2.529 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 9.822 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 0.830 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 1.477 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 6.919 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 4.892 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 9.796 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 0.544 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 0.757 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 1.383 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 1.177 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 0.561 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 0.254 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 0.752 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 18.946 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 1.297 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 16.051 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.903 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 8.313 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.995 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 2.028 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 13.643 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 8.190 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.070 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 31.118 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:12 min
[INFO] Finished at: 2019-08-05T08:14:15Z
[INFO] ------------------------------------------------------------------------
builder@yq-hadoop:~/hadoop$ arch
aarch64
builder@yq-hadoop:~/hadoop$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
{code}
> Hadoop native build failure on ARM hard-float
> ---------------------------------------------
>
> Key: HADOOP-9320
> URL: https://issues.apache.org/jira/browse/HADOOP-9320
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Environment: $ uname -a
> Linux 3.5.0-1000-highbank #154-Ubuntu SMP Thu Jan 10 09:13:40 UTC 2013 armv7l armv7l armv7l GNU/Linux
> $ java -version
> java version "1.8.0-ea"
> Java(TM) SE Runtime Environment (build 1.8.0-ea-b36e)
> Java HotSpot(TM) Client VM (build 25.0-b04, mixed mode)
> Reporter: Trevor Robinson
> Assignee: Trevor Robinson
> Priority: Major
> Labels: BB2015-05-TBR, build-failure
> Attachments: HADOOP-9320.patch
>
>
> ARM JVM float ABI detection is failing in JNIFlags.cmake because JAVA_JVM_LIBRARY is not set at that point. The failure currently causes CMake to assume a soft-float JVM. This causes the build to fail with hard-float OpenJDK (but don't use that) and [Oracle Java 8 Preview for ARM|http://jdk8.java.net/fxarmpreview/]. Hopefully the April update of Oracle Java 7 will support hard-float as well.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org