You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Dmitry Chuyko (JIRA)" <ji...@apache.org> on 2017/11/17 09:42:00 UTC
[jira] [Commented] (MAPREDUCE-6980) Add support of ARM64 hardware
crc instructions for nativetask
[ https://issues.apache.org/jira/browse/MAPREDUCE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256737#comment-16256737 ]
Dmitry Chuyko commented on MAPREDUCE-6980:
------------------------------------------
Since Java 9 CRC32C is also available in core class library just like CRC32. Both are backed by platform specific intrinsics that even have diferent implementations depending on hardware capabilities. For ARM64 you can face that current JDK implementation can perform 1.5x better than the one in jni native library:
https://issues.apache.org/jira/browse/HADOOP-15033?focusedCommentId=16256722&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16256722
> Add support of ARM64 hardware crc instructions for nativetask
> -------------------------------------------------------------
>
> Key: MAPREDUCE-6980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6980
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: nativetask, performance
> Affects Versions: 3.0.0-alpha3
> Environment: ARM64 platform with CRC extension
> Reporter: Jun He
> Priority: Minor
> Attachments: MAPREDUCE-6980.0.patch
>
>
> CRC instructions are introduced since ARM64V8 ISA. With this extension Arm supports CRC32(0x04C11DB7) and CRC32C(0x1EDC6F41) checksum hardware computation.
> This patch enables support on such arm64 platform, similiar as [HADOOP-11660|https://issues.apache.org/jira/browse/HADOOP-11660].
> Benchmark is done on a A57 platform using nttest (hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/test/nttest), and the result is as following:
> Before (sw crc):
> [ RUN ] Perf.CRC
> 17/10/10 03:05:18 INFO CRC time: 0.10112s size: 100.000M speed: 988.92M/s
> 17/10/10 03:05:18 INFO CRC32C time: 0.10043s size: 100.000M speed: 995.74M/s
> [ OK ] Perf.CRC (216 ms)
> After (hw crc):
> [ RUN ] Perf.CRC
> 17/10/11 05:06:27 INFO CRC time: 0.03173s size: 100.000M speed: 3151.14M/s
> 17/10/11 05:06:27 INFO CRC32C time: 0.03146s size: 100.000M speed: 3178.97M/s
> [ OK ] Perf.CRC (77 ms)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org