You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Akira Ajisaka (Jira)" <ji...@apache.org> on 2020/02/14 06:09:00 UTC
[jira] [Assigned] (HADOOP-16850) Support getting thread info from
thread group for JvmMetrics to improve the performance
[ https://issues.apache.org/jira/browse/HADOOP-16850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Akira Ajisaka reassigned HADOOP-16850:
--------------------------------------
Assignee: Tao Yang
> Support getting thread info from thread group for JvmMetrics to improve the performance
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-16850
> URL: https://issues.apache.org/jira/browse/HADOOP-16850
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
> Affects Versions: 2.8.6, 2.9.3, 3.1.4, 3.2.2, 2.10.1, 3.3.1
> Reporter: Tao Yang
> Assignee: Tao Yang
> Priority: Major
> Attachments: HADOOP-16850.001.patch, HADOOP-16850.002.patch
>
>
> Recently we found jmx request taken almost 5s+ to be done when there were 1w+ threads in a stressed datanode process, meanwhile other http requests were blocked and some disk operations were affected (we can see many "Slow manageWriterOsCache" messages in DN log, and these messages were hard to be seen again after we stopped sending jxm requests)
> The excessive time is spent in getting thread info via ThreadMXBean inside which ThreadImpl#getThreadInfo native method is called, the time complexity of ThreadImpl#getThreadInfo is O(n*n) according to [JDK-8185005|https://bugs.openjdk.java.net/browse/JDK-8185005] and it holds global thread lock and prevents creation or termination of threads.
> To improve this, I propose to support getting thread info from thread group which will improve a lot by default, also support using original approach when "-Dhadoop.metrics.jvm.use-thread-mxbean=true" is configured in the startup command.
> An example of performance tests between these two approaches is as follows:
> {noformat}
> #Threads=100, ThreadMXBean=382372 ns, ThreadGroup=72046 ns, ratio: 5
> #Threads=200, ThreadMXBean=776619 ns, ThreadGroup=83875 ns, ratio: 9
> #Threads=500, ThreadMXBean=3392954 ns, ThreadGroup=216269 ns, ratio: 15
> #Threads=1000, ThreadMXBean=9475768 ns, ThreadGroup=220447 ns, ratio: 42
> #Threads=2000, ThreadMXBean=53833729 ns, ThreadGroup=579608 ns, ratio: 92
> #Threads=3000, ThreadMXBean=196829971 ns, ThreadGroup=1157670 ns, ratio: 170
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org