You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2010/01/13 03:47:54 UTC
[jira] Created: (HADOOP-6490) Path.normalize should use
StringUtils.replace in favor of String.replace
Path.normalize should use StringUtils.replace in favor of String.replace
------------------------------------------------------------------------
Key: HADOOP-6490
URL: https://issues.apache.org/jira/browse/HADOOP-6490
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.1
Reporter: Zheng Shao
in our environment, we are seeing that the JobClient is going out of memory because Path.normalizePath(String) is called several tens of thousands of times, and each time it calls "String.replace" twice.
java.lang.String.replace compiles a regex to do the job which is very costly.
We should use org.apache.commons.lang.StringUtils.replace which is much faster and consumes almost no extra memory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6490) Path.normalize should use
StringUtils.replace in favor of String.replace
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799606#action_12799606 ]
Zheng Shao commented on HADOOP-6490:
------------------------------------
Constructor {{Path@Path(Path, Path)}} calls {{normalizePath}}
> Path.normalize should use StringUtils.replace in favor of String.replace
> ------------------------------------------------------------------------
>
> Key: HADOOP-6490
> URL: https://issues.apache.org/jira/browse/HADOOP-6490
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: Zheng Shao
>
> in our environment, we are seeing that the JobClient is going out of memory because Path.normalizePath(String) is called several tens of thousands of times, and each time it calls "String.replace" twice.
> java.lang.String.replace compiles a regex to do the job which is very costly.
> We should use org.apache.commons.lang.StringUtils.replace which is much faster and consumes almost no extra memory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6490) Path.normalize should use
StringUtils.replace in favor of String.replace
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799548#action_12799548 ]
Eli Collins commented on HADOOP-6490:
-------------------------------------
I noticed Path#normalizePath and Shard#normalizePath have diverged, perhaps worth merging them while your at it so you fix the performance issue in both places. Sounds fishy that normalizePath is called tens of thousands of times, perhaps not necessary or could cache the result if it's being called frequently on the same path.
> Path.normalize should use StringUtils.replace in favor of String.replace
> ------------------------------------------------------------------------
>
> Key: HADOOP-6490
> URL: https://issues.apache.org/jira/browse/HADOOP-6490
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.1
> Reporter: Zheng Shao
>
> in our environment, we are seeing that the JobClient is going out of memory because Path.normalizePath(String) is called several tens of thousands of times, and each time it calls "String.replace" twice.
> java.lang.String.replace compiles a regex to do the job which is very costly.
> We should use org.apache.commons.lang.StringUtils.replace which is much faster and consumes almost no extra memory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.