You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by Wei-Chiu Chuang <we...@cloudera.com.INVALID> on 2021/05/18 23:44:00 UTC

Hadoop LZ4 codec questions

Hi I'm trying to understand the LZ4 codec usage in Hadoop.

Liang-Chi replaced the LZ4 codec with the lz4-java HADOOP-17292. The intent
is so that we can use the native library that is bundled in the jar, no
need to install lz4 native libraries on the host machine.

However, there's another LZ4 codec that we ship inside
hadoop-mapreduce-client-nativetask.
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/lz4/lz4.c

What's the purpose of this file? Does the MapReduce client use a different
path to do lz4 compression? Maybe it's deadcode?