[jira] [Commented] (BIGTOP-3535) Build libhadoop.so with zstd

["Masatake Iwasaki (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322548#comment-17322548 ] 

Masatake Iwasaki commented on BIGTOP-3535:
------------------------------------------

[~xyu] Based on the discussion in BIGTOP-3471, we are going to support following ditros in the next release.
 * CentOS 7, 8
 * Debian 10, 11 (if officially released before this release)
 * Fedora 33
 * Ubuntu 18.04, 20.04

You need to update Puppet manifests of [bigtop_toolchain|https://github.com/apache/bigtop/tree/master/bigtop_toolchain] to set up zstd libs on those platforms. If one nof them does not provide zstd package, installing from source code could be an option.

I think zstd feature is enabled if zstd libs/headers are found in default search path even without {{-Drequire.zstd}}. If you want to make build fail without zstd, you can add the option to MAVEN_OPTS of [do-component-build of Hadoop|https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build].

You also need to consider add dependency of packages or not.
 * [https://github.com/apache/bigtop/blob/master/bigtop-packages/src/rpm/hadoop/SPECS/hadoop.spec#L184-L186]
 * [https://github.com/apache/bigtop/blob/master/bigtop-packages/src/deb/hadoop/control#L26]

> Build libhadoop.so with zstd
> ----------------------------
>
>                 Key: BIGTOP-3535
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-3535
>             Project: Bigtop
>          Issue Type: Bug
>          Components: hadoop
>    Affects Versions: 1.5.0
>            Reporter: Xiao Yu
>            Priority: Minor
>
> [Zstandard compression|https://github.com/facebook/zstd] support was added to Hadoop 2.9.0 with HADOOP-13578 it would be great if we bundled support for it with native lib. As it stands now with Bigtop 1.5.0 we don't support it:
> {code}
> $ hadoop checknative -a
> 21/04/13 23:18:20 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
> 21/04/13 23:18:20 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
> Native library checking:
> hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
> zlib:    true /lib/x86_64-linux-gnu/libz.so.1
> snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
> zstd  :  false
> lz4:     true revision:10301
> bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
> openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
> 21/04/13 23:18:20 INFO util.ExitUtil: Exiting with status 1: ExitException
> {code}
> At least with Debian, after installing headers for {{zstd}} with {{apt-get install libzstd-dev}} and rebuilding the native lib with {{mvn package -Pdist,native -DskipTests -Drequire.zstd ...}} support appears to be enabled:
> {code}
> $ hadoop checknative -a
> 21/04/13 23:25:46 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
> 21/04/13 23:25:46 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
> Native library checking:
> hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
> zlib:    true /lib/x86_64-linux-gnu/libz.so.1
> snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
> zstd  :  true /usr/lib/x86_64-linux-gnu/libzstd.so.1
> lz4:     true revision:10301
> bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
> openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)