You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by queiy <12...@163.com> on 2021/08/06 02:47:42 UTC
hadoop3.1.4 EC enable isa-l performance is not improved
hi, everyone
I install hadoop3.1.4 cluster with 3 datanode and HA namenode , A directory ErasureCodingPolicy used XOR-2-1-1024k， but I put a 14G file to the EC directory for test，I found enable isa-l no performance improvement. In two scenarios put a 14G file to directory Consume about 55s. I am sure the isa-l is loaded in my hadoop cluster。such as "hadoop checknative" command output :
root@node-06:/# hadoop checknative
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
2021-08-06 10:43:28,874 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2021-08-06 10:43:28,880 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
2021-08-06 10:43:28,949 INFO nativeio.NativeIO: The native code was built with PMDK support, and PMDK libs were loaded successfully.
Native library checking:
hadoop: true /opt/hadoop-3.1.4/lib/native/libhadoop.so.1.0.0
zlib: true /lib/x86_64-linux-gnu/libz.so.1
zstd : true /opt/hadoop-3.1.4/lib/native/libzstd.so.1
snappy: true /opt/hadoop-3.1.4/lib/native/libsnappy.so.1
lz4: true revision:10301
bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L: true /opt/hadoop-3.1.4/lib/native/libisal.so.2
PMDK: true /usr/local/lib/libpmem.so.1.0.0
What’s the problem, Can anyone help me ?
Re: hadoop3.1.4 EC enable isa-l performance is not improved
Posted by Bob Metelsky <bo...@pm.me.INVALID>.
Likely try
yum install openssl
https://www.linuxquestions.org/questions/linux-software-2/need-libcrypto-so-6-64bit-and-libssl-so-6-64bit-for-redhat-6-a-873068/
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, August 5th, 2021 at 10:47 PM, queiy <12...@163.com> wrote:
> hi, everyone
> I install hadoop3.1.4 cluster with 3 datanode and HA namenode , A directory ErasureCodingPolicy used XOR-2-1-1024k， but I put a 14G file to the EC directory for test，I found enable isa-l no performance improvement. In two scenarios put a 14G file to directory Consume about 55s. I am sure the isa-l is loaded in my hadoop cluster。such as "hadoop checknative" command output :
>
> root@node-06:/# hadoop checknative
> WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
> 2021-08-06 10:43:28,874 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
> 2021-08-06 10:43:28,880 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
> 2021-08-06 10:43:28,949 INFO nativeio.NativeIO: The native code was built with PMDK support, and PMDK libs were loaded successfully.
> Native library checking:
> hadoop: true /opt/hadoop-3.1.4/lib/native/libhadoop.so.1.0.0
> zlib: true /lib/x86_64-linux-gnu/libz.so.1
> zstd : true /opt/hadoop-3.1.4/lib/native/libzstd.so.1
> snappy: true /opt/hadoop-3.1.4/lib/native/libsnappy.so.1
> lz4: true revision:10301
> bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1
> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
> ISA-L: true /opt/hadoop-3.1.4/lib/native/libisal.so.2
> PMDK: true /usr/local/lib/libpmem.so.1.0.0
>
> What’s the problem, Can anyone help me ?
>
> [“已经过社区验证”图标]