You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Prasanth Jayachandran (JIRA)" <ji...@apache.org> on 2018/07/11 08:57:00 UTC
[jira] [Commented] (HADOOP-15598) DataChecksum calculate checksum
is contented on hashtable synchronization
[ https://issues.apache.org/jira/browse/HADOOP-15598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539750#comment-16539750 ]
Prasanth Jayachandran commented on HADOOP-15598:
------------------------------------------------
Results from JMH microbenchmark. As we can see for bytebuffers of size >128 there is no significant benefit as most of the time is spent on the chunk looping. But for smaller size, the synchronization has significant impact.
{noformat}
Benchmark (size) Mode Cnt Score Error Units
ComputeChecksumBenchmark.benchmarkChecksumWithoutPatch 8 avgt 10 66.970 ± 1.247 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithoutPatch 16 avgt 10 91.524 ± 2.141 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithoutPatch 32 avgt 10 127.155 ± 8.037 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithoutPatch 64 avgt 10 233.941 ± 14.327 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithoutPatch 128 avgt 10 382.898 ± 4.316 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithoutPatch 1024 avgt 10 2707.026 ± 53.266 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithoutPatch 4096 avgt 10 11411.330 ± 181.900 ns/op
Benchmark (size) Mode Cnt Score Error Units
ComputeChecksumBenchmark.benchmarkChecksumWithPatch 8 avgt 10 27.948 ± 0.507 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithPatch 16 avgt 10 49.336 ± 1.212 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithPatch 32 avgt 10 96.344 ± 1.360 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithPatch 64 avgt 10 182.513 ± 2.553 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithPatch 128 avgt 10 356.205 ± 4.282 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithPatch 1024 avgt 10 2825.526 ± 31.177 ns/op
ComputeChecksumBenchmark.benchmarkChecksumWithPatch 4096 avgt 10 12078.055 ± 262.343 ns/op
{noformat}
> DataChecksum calculate checksum is contented on hashtable synchronization
> -------------------------------------------------------------------------
>
> Key: HADOOP-15598
> URL: https://issues.apache.org/jira/browse/HADOOP-15598
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 3.2.0
> Reporter: Prasanth Jayachandran
> Priority: Major
> Attachments: HADOOP-15598.1.patch, Screen Shot 2018-07-11 at 1.45.06 AM.png
>
>
> When profiling a multi-threaded hive streaming ingest, observed lock contention on java.util.Properties getProperty() to check if os is "sparc". java.util.Properties internally uses HashTable. HashTable.get() is synchronized method. In the test application, on a 30s profile with 64 threads ~40% CPU time is spent on getProperty() contention. See attached snapshot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org