You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2018/06/20 19:15:00 UTC
[jira] [Commented] (HADOOP-15550) Avoid static initialization of
ObjectMappers
[ https://issues.apache.org/jira/browse/HADOOP-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518508#comment-16518508 ]
Todd Lipcon commented on HADOOP-15550:
--------------------------------------
Benchmarked with a simple program that does {{new Path("/").getFileSystem(new Configuration());}}
The attached patch avoids loading about 400 classes, and saves some measurable CPU:
{code}
without patch (2219 classes loaded):
1378.393961 task-clock (msec) # 1.959 CPUs utilized ( +- 0.57% )
2,076 context-switches # 0.002 M/sec ( +- 0.62% )
45 cpu-migrations # 0.033 K/sec ( +- 3.75% )
30,529 page-faults # 0.022 M/sec ( +- 0.24% )
4,540,069,263 cycles # 3.294 GHz ( +- 0.92% )
5,282,002,987 instructions # 1.16 insn per cycle ( +- 0.93% )
991,080,821 branches # 719.011 M/sec ( +- 0.90% )
40,313,544 branch-misses # 4.07% of all branches ( +- 0.67% )
0.703624736 seconds time elapsed ( +- 0.72% )
with patch (1821 classes loaded):
1269.949263 task-clock (msec) # 2.082 CPUs utilized ( +- 1.11% )
2,008 context-switches # 0.002 M/sec ( +- 0.76% )
51 cpu-migrations # 0.040 K/sec ( +- 8.14% )
25,034 page-faults # 0.020 M/sec ( +- 0.26% )
4,157,369,649 cycles # 3.274 GHz ( +- 0.78% )
4,674,086,838 instructions # 1.12 insn per cycle ( +- 0.42% )
870,359,803 branches # 685.350 M/sec ( +- 0.41% )
36,028,258 branch-misses # 4.14% of all branches ( +- 0.44% )
0.610038881 seconds time elapsed ( +- 1.54% )
{code}
> Avoid static initialization of ObjectMappers
> --------------------------------------------
>
> Key: HADOOP-15550
> URL: https://issues.apache.org/jira/browse/HADOOP-15550
> Project: Hadoop Common
> Issue Type: Bug
> Components: performance
> Affects Versions: 3.2.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: hadoop-15550.txt
>
>
> Various classes statically initialize an ObjectMapper READER instance. This ends up doing a bunch of class-loading of Jackson libraries that can add up to a fair amount of CPU, even if the reader ends up not being used. This is particularly the case with WebHdfsFileSystem, which is class-loaded by a serviceloader even when unused in a particular job. We should lazy-init these members instead of doing so as a static class member.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org