[jira] [Created] (BIGTOP-3535) Build libhadoop.so with zstd

["Xiao Yu (Jira)" <ji...@apache.org>]

Xiao Yu created BIGTOP-3535:
-------------------------------

             Summary: Build libhadoop.so with zstd
                 Key: BIGTOP-3535
                 URL: https://issues.apache.org/jira/browse/BIGTOP-3535
             Project: Bigtop
          Issue Type: Bug
          Components: hadoop
    Affects Versions: 1.5.0
            Reporter: Xiao Yu


[Zstandard compression|https://github.com/facebook/zstd] support was added to Hadoop 2.9.0 with HADOOP-13578 it would be great if we bundled support for it with native lib. As it stands now with Bigtop 1.5.0 we don't support it:

{code}
$ hadoop checknative -a
21/04/13 23:18:20 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
21/04/13 23:18:20 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
zstd  :  false
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
21/04/13 23:18:20 INFO util.ExitUtil: Exiting with status 1: ExitException
{code}

At least with Debian, after installing headers for {{zstd}} with {{apt-get install libzstd-dev}} and rebuilding the native lib with {{mvn package -Pdist,native -DskipTests -Drequire.zstd ...}} support appears to be enabled:

{code}
$ hadoop checknative -a
21/04/13 23:25:46 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
21/04/13 23:25:46 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
zstd  :  true /usr/lib/x86_64-linux-gnu/libzstd.so.1
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)