You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Barnabas Maidics (JIRA)" <ji...@apache.org> on 2018/07/20 21:35:00 UTC
[jira] [Comment Edited] (HDFS-13752) fs.Path stores file path in
java.net.URI causes big memory waste
[ https://issues.apache.org/jira/browse/HDFS-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551317#comment-16551317 ]
Barnabas Maidics edited comment on HDFS-13752 at 7/20/18 9:34 PM:
------------------------------------------------------------------
Hi [~misha@cloudera.com],
I used Hive 2.1.1 and hadoop 2.7.6 locally and used VisualVM to create the heap dump and for analysis as well. I can't attach the original heap dump because of its size, but I've uploaded it on drive:
[https://drive.google.com/open?id=167aLQRSW2LW2qrqr6P5UcwNqSPSCCQkN]
You're right that the 42% is the retained memory, but even the URI-s itself take 8.1%. I worked with 100k partitions (with only ~20 row in each) which may not be the most realistic scenario but I was measuring the effect of increasing the partition numbers.
was (Author: b.maidics):
Hi [~misha@cloudera.com],
I used Hive 2.1.1 and hadoop 2.7.6 locally and used VisualVM to create the heap dump and for analysis as well. I can't attach the original heap dump because of its size, but I've uploaded it on drive:
https://drive.google.com/open?id=167aLQRSW2LW2qrqr6P5UcwNqSPSCCQkN
You're right that the 42% is the retained memory, but even the URI-s itself take 8.1%. I worked with 100k partitions which may not be the most realistic scenario but I was measuring the effect of increasing the partition numbers.
> fs.Path stores file path in java.net.URI causes big memory waste
> ----------------------------------------------------------------
>
> Key: HDFS-13752
> URL: https://issues.apache.org/jira/browse/HDFS-13752
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: fs
> Reporter: Barnabas Maidics
> Priority: Major
> Attachments: Screen Shot 2018-07-20 at 11.12.38.png
>
>
> I was looking at HiveServer2 memory usage, and a big percentage of this was because of org.apache.hadoop.fs.Path, where you store file paths in a java.net.URI object. The URI implementation stores the same string in 3 different objects (see the attached image). In Hive when there are many partitions this cause a big memory usage. In my particular case 42% of memory was used by java.net.URI so it could be reduced to 14%.
> I wonder if the community is open to replace it with a more memory efficient implementation and what other things should be considered here? It can be a huge memory improvement for Hadoop and for Hive as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org