unsubscribe

[an...@yahoo.com.INVALID]

 

 

Da: Andrew Purtell <an...@gmail.com> 
Inviato: giovedì 24 marzo 2022 23:14
A: Joe Mocker <jm...@magnite.com.invalid>
Cc: user@hadoop.apache.org
Oggetto: Re: Building Hadoop on macOS Monterey?

 

If you build with -Dbundle.snappy -Dbundle.zstd on the Maven command line this would produce a tarball containing copies of the native shared libraries in lib/native/ and this would be like your symlink workaround but perhaps less hacky and something the build supports already. Does this work for you? 





On Mar 19, 2022, at 8:09 AM, Joe Mocker <jmocker@magnite.com.invalid <ma...@magnite.com.invalid> > wrote:

Hi,

 

Curious if anyone has tips for building Hadoop on macOS Monterey, for Apple Silicon? My goal is to be able to use native (compression) libraries. After some gymnastics, I have been able to compile Hadoop 2.9.1 but problems arise locating and loading dynamic libraries.

 

For example running hadoop checknative results in the following

 

22/03/19 07:57:00 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version

22/03/19 07:57:00 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library

22/03/19 07:57:00 ERROR snappy.SnappyCompressor: failed to load SnappyCompressor

java.lang.UnsatisfiedLinkError: Cannot load libsnappy.1.dylib (dlopen(libsnappy.1.dylib, 0x0009): tried: '/Volumes/work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64/zulu-8.jdk/Contents/Home/bin/./libsnappy.1.dylib' (no such file), 'libsnappy.1.dylib' (no such file), '/usr/lib/libsnappy.1.dylib' (no such file), '/Volumes/work/hadoop-2.9.1/libsnappy.1.dylib' (no such file))!

                  at org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs(Native Method)

                  at org.apache.hadoop.io.compress.snappy.SnappyCompressor.<clinit>(SnappyCompressor.java:57)

                  at org.apache.hadoop.io.compress.SnappyCodec.isNativeCodeLoaded(SnappyCodec.java:82)

                  at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:92)

22/03/19 07:57:00 WARN zstd.ZStandardCompressor: Error loading zstandard native libraries: java.lang.InternalError: Cannot load libzstd.1.dylib (dlopen(libzstd.1.dylib, 0x0009): tried: '/Volumes/work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64/zulu-8.jdk/Contents/Home/bin/./libzstd.1.dylib' (no such file), 'libzstd.1.dylib' (no such file), '/usr/lib/libzstd.1.dylib' (no such file), '/Volumes/work/hadoop-2.9.1/libzstd.1.dylib' (no such file))!

WARNING: /work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64//bin/java is loading libcrypto in an unsafe way

Abort trap: 6

 

No matter what combination I try of setting LD_LIBRARY_PATH, DYLD_LIBRARY_PATH and/or DYLD_FALLBACK_LIBRARY_PATH it will not find the necessary libraries. I think this has to do with restrictions due to Apple’s System Integrity Protection (SIP).

 

The only way I have figured out how to work around this so far is to symlink all the dynamic libraries in one location then run hadoop from that working directory, for example

 

lrwxrwxr-x  1 mock  staff     59 Mar 18 17:55 libcrypto.dylib@ -> /opt/homebrew/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.dylib <mailto:/opt/homebrew/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.dylib> 

lrwxrwxr-x  1 mock  staff     45 Mar 18 18:09 libhadoop.dylib@ -> /work/hadoop-2.9.1/lib/native/libhadoop.dylib

lrwxrwxr-x  1 mock  staff     53 Mar 18 17:55 libsnappy.1.dylib@ -> /opt/homebrew/Cellar/snappy/1.1.9/lib/libsnappy.dylib

lrwxrwxr-x  1 mock  staff     51 Mar 18 18:05 libzstd.1.dylib@ -> /opt/homebrew/Cellar/zstd/1.5.2/lib/libzstd.1.dylib





% $HADOOP_HOME/bin/hadoop checknative

22/03/19 08:05:55 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version

22/03/19 08:05:55 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library

Native library checking:

hadoop:  true /Volumes/work/hadoop-2.9.1/lib/native/libhadoop.dylib

zlib:    true /usr/lib/libz.1.dylib

snappy:  true /opt/homebrew/Cellar/snappy/1.1.9/lib/libsnappy.1.1.9.dylib

zstd  :  true /opt/homebrew/Cellar/zstd/1.5.2/lib/libzstd.1.5.2.dylib

lz4:     true revision:10301

bzip2:   false 

openssl: false EVP_CIPHER_CTX_cleanup





What am really looking to do is use Spark (and Jupyter), with native libraries, which adds even more wrinkles to it.





Any suggestions would be appreciated.

 

  —joe