You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org> on 2010/02/02 19:06:19 UTC
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828694#action_12828694 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
The archive command is roughly double the memory usage compared with ls. Below are the jmap outputs on 10^5 files
- hadoop archive
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 665091 43671104 [C
2: 1471573 35317752 java.lang.String
3: 261286 20902880 java.net.URI
4: 261276 16721664 org.apache.hadoop.fs.FileStatus
5: 200001 4800024 org.apache.hadoop.fs.permission.FsPermission
6: 261285 4180560 org.apache.hadoop.fs.Path
{noformat}
- hadoop fs -ls
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 304173 21091592 [C
2: 804254 19302096 java.lang.String
3: 100009 8000720 java.net.URI
4: 100001 6400064 org.apache.hadoop.fs.FileStatus
5: 100002 2400048 org.apache.hadoop.fs.permission.FsPermission
6: 100008 1600128 org.apache.hadoop.fs.Path
{noformat}
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.