[jira] [Commented] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866284#comment-16866284 ] 

Todd Lipcon commented on HDFS-12046:
------------------------------------

What does the assembly look like for the ISAL variiant? Is it just a matter of there being better throughput of the crc32q instruction so we need to unroll it once more than we already do? Can you paste the disassembly of the hot path? It seems like it would be nice to just fix our existing implementation vs requiring another library to be passed around to clients, etc

> Hadoop CRC implementation using Intel ISA-L library
> ---------------------------------------------------
>
>                 Key: HDFS-12046
>                 URL: https://issues.apache.org/jira/browse/HDFS-12046
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: luhuichun
>            Assignee: Sammi Chen
>            Priority: Major
>         Attachments: HDFS-12046-001.patch, ISA-L CRC Performance Report using intel ISA-L.pdf
>
>
> Intel ISA-L open source library provides set of highly optimized functions for RAID, erasure code, CRC, cryptographic hash, encryption, and compression. Ref. https://github.com/01org/isa-l. HDFS-EC has already integrated ISA-L and added the necessary building options support for Hadoop. For Hadoop CRC, we recently explored more, developing a Hadoop CRC using Intel ISA-L, performing a test on Broadwell and Skylake servers, comparing the performance against Hadoop native CRC. On Broadwell/Skylake, ISA-L CRC has about 8%~ performance gain over Hadoop native CRC. We suggest adding a new Hadoop native CRC using the ISA-L library, the extra advantage is it’s already optimized when we upgrade to new servers and Hadoop developers don’t have to maintain their own bunch of ASM codes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org