You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/03/14 09:45:00 UTC
[jira] [Work logged] (HADOOP-18129) Change URI[] in INodeLink to String[] to reduce memory footprint of ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-18129?focusedWorklogId=740754&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-740754 ]
ASF GitHub Bot logged work on HADOOP-18129:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 14/Mar/22 09:44
Start Date: 14/Mar/22 09:44
Worklog Time Spent: 10m
Work Description: abhishekdas99 commented on a change in pull request #3996:
URL: https://github.com/apache/hadoop/pull/3996#discussion_r825754092
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##########
@@ -216,19 +216,19 @@ public boolean equals(Object obj) {
/**
* Array of target FileSystem URIs.
*/
- private final URI[] targetFileSystemURIs;
+ private final String[] targetFileSystemPaths;
- MountPoint(Path srcPath, URI[] targetFs) {
+ MountPoint(Path srcPath, String[] targetFs) {
mountedOnPath = srcPath;
- targetFileSystemURIs = targetFs;
+ targetFileSystemPaths = targetFs;
}
public Path getMountedOnPath() {
return mountedOnPath;
}
- public URI[] getTargetFileSystemURIs() {
- return targetFileSystemURIs;
+ public String[] getTargetFileSystemPaths() {
Review comment:
Added getTargetFileSystemURIs where we are returning URI[]. Also added method to return targetFileSystem Paths.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##########
@@ -294,11 +294,11 @@ void addLink(final String pathComponent, final INodeLink<T> link)
*/
INodeLink(final String pathToNode, final UserGroupInformation aUgi,
Function<URI, T> createFileSystemMethod,
- final URI aTargetDirLink) {
+ final String aTargetDirLink) throws URISyntaxException {
super(pathToNode, aUgi);
targetFileSystem = null;
- targetDirLinkList = new URI[1];
- targetDirLinkList[0] = aTargetDirLink;
+ targetDirLinkList = new String[1];
+ targetDirLinkList[0] = new URI(aTargetDirLink).toString();
Review comment:
In the previous implementation, we were creating the URI object from the target file system path. If the path is not a valid URI, the filesystem initialization should have failed. As we are not keeping the URI object anymore, this ensures we are not dealing with an invalid URI and new URI object validates the target filesystem path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 740754)
Time Spent: 50m (was: 40m)
> Change URI[] in INodeLink to String[] to reduce memory footprint of ViewFileSystem
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-18129
> URL: https://issues.apache.org/jira/browse/HADOOP-18129
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Abhishek Das
> Assignee: Abhishek Das
> Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Around 40k instances of INodeLink each of which is taking memory ranging from 1680bytes to 1160bytes. Multiplying 40k with 1160bytes will give us approximate 45mb.
> With changing from URI to String in INodeLink the memory consumed by each of the INodeLink objects has reduced from ~1160 bytes to ~320 bytes. Overall size becomes (40k X 320) 12mb
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org