[jira] [Commented] (HBASE-8323) Low hanging checksum improvements

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13667167#comment-13667167 ] 

Todd Lipcon commented on HBASE-8323:
------------------------------------

bq. Enable NativeCrc32 to be used as a checksum algo. It is not clear how much gain we can expect over pure java CRC32.

The gain's really big -- something around 10x CPU savings. Obviously that doesn't turn into a 10x improvement of HBase throughput, but I bet it would be substantial.
                
> Low hanging checksum improvements
> ---------------------------------
>
>                 Key: HBASE-8323
>                 URL: https://issues.apache.org/jira/browse/HBASE-8323
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Enis Soztutar
>
> Over at Hadoop land, [~tlipcon] had done some improvements for checksums, a native implementation for CRC32C (HADOOP-7445) and bulk verify of checksums (HADOOP-7444). 
> In HBase, we can do
>  - Also develop a bulk verify API. Regardless of hbase.hstore.bytes.per.checksum we always want to verify of the whole checksum for the hfile block.
>  - Enable NativeCrc32 to be used as a checksum algo. It is not clear how much gain we can expect over pure java CRC32. 
> Though, longer term we should focus on convincing hdfs guys for inline checksums (HDFS-2699)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira