You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2021/04/19 20:51:22 UTC
[GitHub] [bigtop] xyu opened a new pull request #767: BIGTOP-3535: Add zstd compression headers to add native lib support
xyu opened a new pull request #767:
URL: https://github.com/apache/bigtop/pull/767
Current Bigtop 1.5.0 build env does not have [zstd compression](https://github.com/facebook/zstd) headers so the Hadoop Native Library that's built does not support `zstd`:
```
$ hadoop checknative -a
21/04/13 23:18:20 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
21/04/13 23:18:20 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib/x86_64-linux-gnu/libz.so.1
snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1
zstd : false
lz4: true revision:10301
bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
21/04/13 23:18:20 INFO util.ExitUtil: Exiting with status 1: ExitException
```
Update the toolchain to install headers for zstd which causes this feature to get turned on:
```
$ hadoop checknative -a
21/04/19 20:41:00 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
21/04/19 20:41:00 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib/x86_64-linux-gnu/libz.so.1
snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1
zstd : true /usr/lib/x86_64-linux-gnu/libzstd.so.1
lz4: true revision:10301
bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
```
I have verified that these headers exist for all the supported distributions:
```
$ docker run -it --rm centos:7 sh -c 'rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm >/dev/null 2>&1 ; yum search libzstd-devel 2>/dev/null'
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
epel/x86_64/metalink | 7.7 kB 00:00
* base: linux-mirrors.fnal.gov
* epel: mirror.team-cymru.com
* extras: repo.ialab.dsu.edu
* updates: mirrors.unifiedlayer.com
base | 3.6 kB 00:00
epel | 4.7 kB 00:00
extras | 2.9 kB 00:00
updates | 2.9 kB 00:00
(1/7): base/7/x86_64/group_gz | 153 kB 00:00
(2/7): extras/7/x86_64/primary_db | 232 kB 00:00
(3/7): epel/x86_64/group_gz | 96 kB 00:01
(4/7): epel/x86_64/updateinfo | 1.0 MB 00:01
(5/7): base/7/x86_64/primary_db | 6.1 MB 00:01
(6/7): epel/x86_64/primary_db | 6.9 MB 00:01
(7/7): updates/7/x86_64/primary_db | 7.1 MB 00:01
========================== N/S matched: libzstd-devel ==========================
libzstd-devel.x86_64 : Header files for Zstd library
Name and summary matches only, use "search all" for everything.
```
```
$ docker run -it --rm centos:8 sh -c 'yum search libzstd-devel 2>/dev/null'
===================== Name Exactly Matched: libzstd-devel ======================
libzstd-devel.i686 : Header files for Zstd library
libzstd-devel.x86_64 : Header files for Zstd library
```
```
$ docker run -it --rm debian:10-slim sh -c 'apt-get update > /dev/null ; apt-cache search libzstd-dev'
libzstd-dev - fast lossless compression algorithm -- development files
```
```
$ docker run -it --rm fedora:33 sh -c 'yum search libzstd-devel 2>/dev/null'
===================== Name Exactly Matched: libzstd-devel ======================
libzstd-devel.i686 : Header files for Zstd library
libzstd-devel.x86_64 : Header files for Zstd library
```
```
$ docker run -it --rm ubuntu:18.04 sh -c 'apt-get update > /dev/null ; apt-cache search libzstd-dev'
libzstd-dev - fast lossless compression algorithm -- development files
libzstd1-dev - transitional package for libzstd-dev
```
```
$ docker run -it --rm ubuntu:20.04 sh -c 'apt-get update > /dev/null ; apt-cache search libzstd-dev'
libzstd-dev - fast lossless compression algorithm -- development files
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims merged pull request #767: BIGTOP-3535: Add zstd compression headers to add native lib support
Posted by GitBox <gi...@apache.org>.
iwasakims merged pull request #767:
URL: https://github.com/apache/bigtop/pull/767
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org