You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "LiXin Ge (JIRA)" <ji...@apache.org> on 2017/06/14 09:43:02 UTC
[jira] [Updated] (HADOOP-14424) Add CRC32C performance test.
[ https://issues.apache.org/jira/browse/HADOOP-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
LiXin Ge updated HADOOP-14424:
------------------------------
Attachment: HADOOP-14424.002.patch
> Add CRC32C performance test.
> ----------------------------
>
> Key: HADOOP-14424
> URL: https://issues.apache.org/jira/browse/HADOOP-14424
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
> Affects Versions: 3.0.0-alpha2
> Reporter: LiXin Ge
> Assignee: LiXin Ge
> Priority: Minor
> Labels: test
> Attachments: HADOOP-14424.002.patch, HADOOP-14424.patch
>
>
> The default checksum algorithm of Hadoop is CRC32C, so we'd better add a new test to compare Crc32C chunked verification implementations.
> This test is based on Crc32PerformanceTest, what I have done in this test is:
> 1.CRC32C performance test.
> 2.CRC32C is not supported by java.util.zip in JAVA JDK, so just remove it from this test.
> 3.User can choose either direct buffer or non-directly buffer to run this test manually.
> 4.Take use of verifyChunkedSumsByteArray for native to support non-directly native test.
> The test result in my environment is:
> [root@master bin]# ./hadoop org.apache.hadoop.util.Crc32CPerformanceTest
> java.version = 1.8.0_111
> java.runtime.name = Java(TM) SE Runtime Environment
> java.runtime.version = 1.8.0_111-b14
> java.vm.version = 25.111-b14
> java.vm.vendor = Oracle Corporation
> java.vm.name = Java HotSpot(TM) 64-Bit Server VM
> java.vm.specification.version = 1.8
> java.specification.version = 1.8
> os.arch = amd64
> os.name = Linux
> os.version = 2.6.33.20
> Data Length = 64 MB
> Trials = 5
> Direct Buffer Performance Table (bpc: byte-per-crc in MB/sec; #T: #Theads)
> | bpc | #T || PureJava || Native | % diff |
> | 32 | 1 | 394.0 | 4156.2 | 954.9% |
> | 32 | 2 | 400.5 | 3679.7 | 818.7% |
> | 32 | 4 | 401.8 | 2657.0 | 561.3% |
> | 32 | 8 | 389.1 | 1633.8 | 319.9% |
> | 32 | 16 | 222.2 | 1116.3 | 402.5% |
> | bpc | #T || PureJava || Native | % diff |
> | 64 | 1 | 465.0 | 5931.0 | 1175.5% |
> | 64 | 2 | 468.8 | 1839.2 | 292.3% |
> | 64 | 4 | 460.4 | 2968.3 | 544.7% |
> | 64 | 8 | 452.4 | 1925.7 | 325.6% |
> | 64 | 16 | 246.9 | 1291.8 | 423.3% |
> | bpc | #T || PureJava || Native | % diff |
> | 128 | 1 | 522.0 | 6147.8 | 1077.6% |
> | 128 | 2 | 366.0 | 4758.5 | 1200.2% |
> | 128 | 4 | 307.8 | 3265.1 | 960.8% |
> | 128 | 8 | 283.6 | 2092.2 | 637.6% |
> | 128 | 16 | 219.9 | 1226.1 | 457.6% |
> | bpc | #T || PureJava || Native | % diff |
> | 256 | 1 | 550.7 | 3177.6 | 477.0% |
> | 256 | 2 | 538.6 | 1933.2 | 258.9% |
> | 256 | 4 | 427.2 | 3278.1 | 667.3% |
> | 256 | 8 | 420.8 | 2272.3 | 440.0% |
> | 256 | 16 | 294.0 | 1311.2 | 346.0% |
> | bpc | #T || PureJava || Native | % diff |
> | 512 | 1 | 553.4 | 3690.4 | 566.9% |
> | 512 | 2 | 455.6 | 4974.1 | 991.7% |
> | 512 | 4 | 494.2 | 3406.4 | 589.2% |
> | 512 | 8 | 431.4 | 2257.0 | 423.2% |
> | 512 | 16 | 316.3 | 1272.0 | 302.2% |
> | bpc | #T || PureJava || Native | % diff |
> | 1024 | 1 | 566.1 | 3520.0 | 521.8% |
> | 1024 | 2 | 508.7 | 4437.4 | 772.3% |
> | 1024 | 4 | 520.7 | 3422.6 | 557.4% |
> | 1024 | 8 | 501.8 | 2124.7 | 323.4% |
> | 1024 | 16 | 340.6 | 1305.0 | 283.2% |
> | bpc | #T || PureJava || Native | % diff |
> | 2048 | 1 | 535.1 | 5438.5 | 916.4% |
> | 2048 | 2 | 537.3 | 4668.3 | 768.8% |
> | 2048 | 4 | 529.2 | 2417.2 | 356.7% |
> | 2048 | 8 | 485.1 | 2249.8 | 363.8% |
> | 2048 | 16 | 334.3 | 1265.7 | 278.6% |
> | bpc | #T || PureJava || Native | % diff |
> | 4096 | 1 | 563.0 | 7264.0 | 1190.1% |
> | 4096 | 2 | 538.8 | 5681.4 | 954.4% |
> | 4096 | 4 | 528.9 | 3107.6 | 487.5% |
> | 4096 | 8 | 521.8 | 2168.9 | 315.6% |
> | 4096 | 16 | 336.5 | 1256.5 | 273.4% |
> | bpc | #T || PureJava || Native | % diff |
> | 8192 | 1 | 521.1 | 6842.4 | 1213.1% |
> | 8192 | 2 | 543.0 | 5510.2 | 914.8% |
> | 8192 | 4 | 526.1 | 3759.5 | 614.6% |
> | 8192 | 8 | 480.1 | 2254.1 | 369.6% |
> | 8192 | 16 | 336.4 | 1213.3 | 260.6% |
> | bpc | #T || PureJava || Native | % diff |
> | 16384 | 1 | 521.7 | 3695.7 | 608.3% |
> | 16384 | 2 | 540.8 | 5610.5 | 937.4% |
> | 16384 | 4 | 539.1 | 3313.0 | 514.6% |
> | 16384 | 8 | 518.7 | 2198.4 | 323.8% |
> | 16384 | 16 | 337.2 | 1237.8 | 267.1% |
> | bpc | #T || PureJava || Native | % diff |
> | 32768 | 1 | 532.1 | 3696.0 | 594.6% |
> | 32768 | 2 | 541.4 | 4647.4 | 758.4% |
> | 32768 | 4 | 536.6 | 2325.1 | 333.3% |
> | 32768 | 8 | 518.8 | 2036.9 | 292.6% |
> | 32768 | 16 | 337.8 | 1232.0 | 264.7% |
> | bpc | #T || PureJava || Native | % diff |
> | 65536 | 1 | 527.7 | 8240.0 | 1461.4% |
> | 65536 | 2 | 561.9 | 5472.3 | 873.9% |
> | 65536 | 4 | 536.5 | 3725.2 | 594.4% |
> | 65536 | 8 | 482.8 | 2040.2 | 322.5% |
> | 65536 | 16 | 335.9 | 1240.4 | 269.3% |
> Elapsed 77.4s
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org