You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by "He Tianyi (JIRA)" <ji...@apache.org> on 2016/04/26 16:36:13 UTC

[jira] [Created] (HDFS-10331) Use java.util.zip.CRC32 for checksum in java8 or above

He Tianyi created HDFS-10331:
--------------------------------

             Summary: Use java.util.zip.CRC32 for checksum in java8 or above
                 Key: HDFS-10331
                 URL: https://issues.apache.org/jira/browse/HDFS-10331
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: datanode, hdfs, hdfs-client
    Affects Versions: 2.6.0
            Reporter: He Tianyi


In java8, performance of intrinsic CRC32 has been dramatically improved.
See: https://bugs.openjdk.java.net/browse/JDK-7088419

I carried an in-memory benchmark of throughput, on a server with two E5-2630 v2 cpus, results:
java7  java.util.zip.CRC32: 0.81GB/s
hdfs DataChecksum, native: 1.46GB/s
java8  java.util.zip.CRC32: 2.39GB/s
hdfs DataChecksum, CRC32 on java8: 2.39GB/s

IMHO I think we could either:
A) provide a configuration for user to switch CRC32 implementations;
or B) On java8 or above, always use intrinsic CRC32.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)