You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org> on 2010/01/28 23:27:34 UTC
[jira] Created: (MAPREDUCE-1425) archive throws OutOfMemoryError
archive throws OutOfMemoryError
-------------------------------
Key: MAPREDUCE-1425
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Mahadev konar
{noformat}
-bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.regex.Pattern.compile(Pattern.java:1432)
at java.util.regex.Pattern.<init>(Pattern.java:1133)
at java.util.regex.Pattern.compile(Pattern.java:847)
at java.lang.String.replace(String.java:2208)
at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
at org.apache.hadoop.fs.Path.initialize(Path.java:137)
at org.apache.hadoop.fs.Path.<init>(Path.java:126)
at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828872#action_12828872 ]
Mahadev konar commented on MAPREDUCE-1425:
------------------------------------------
the patch is against 0.20 branch of hadoop.
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832644#action_12832644 ]
Hudson commented on MAPREDUCE-1425:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #232 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/232/])
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831626#action_12831626 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
Hudson does not seem working. Ran test-patch locally.
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no new tests are needed for this patch.
[exec] Also please list what manual steps were performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
{noformat}
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1425:
-------------------------------------
Status: Patch Available (was: Open)
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1425:
----------------------------------------------
Issue Type: Improvement (was: Bug)
Hadoop Flags: [Reviewed]
+1 patch looks good.
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829882#action_12829882 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
With the patch + MAPREDUCE-1399
{noformat}
$hadoop archive -Dmapred.job.queue.name=q -archiveName t300.har -p /user/tsz t300 /user/tsz
Exception in archives
java.lang.NullPointerException
at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:662)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:681)
{noformat}
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1425:
----------------------------------------------
Component/s: harchive
Fix Version/s: 0.22.0
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1425:
-------------------------------------
Attachment: MAPREDUCE-1425.patch
sorry the name of the file should have been mapreduce and not zookeeper patch!! Looks like I am not good at multi tasking!
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806135#action_12806135 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
Got NPE this time.
{noformat}
-bash-3.1$ hadoop archive -archiveName t4.har -p /user/tsz t4 .
Exception in thread "LeaseChecker" 10/01/28 22:33:17 WARN hdfs.DFSClient: DataStreamer Exception: java.lang.NullPointerException
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:467)
at org.apache.hadoop.ipc.Client.call(Client.java:720)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
10/01/28 22:33:17 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
10/01/28 22:33:17 WARN hdfs.DFSClient: Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_cezmmo/_har_src_files" - Aborting...
Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_cezmmo/_har_src_files" - Aborting...
{noformat}
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830207#action_12830207 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
After the patch, there are some improvement but archive still uses double memory of ls.
- archive
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 658875 42680832 [C
2: 1434792 34435008 java.lang.String
3: 255174 20413920 java.net.URI
4: 255163 16330432 org.apache.hadoop.fs.FileStatus
5: 200001 4800024 org.apache.hadoop.fs.permission.FsPermission
6: 255172 4082752 org.apache.hadoop.fs.Path
{noformat}
- ls
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 304186 21086344 [C
2: 804264 19302336 java.lang.String
3: 100009 8000720 java.net.URI
4: 100001 6400064 org.apache.hadoop.fs.FileStatus
5: 100002 2400048 org.apache.hadoop.fs.permission.FsPermission
6: 100008 1600128 org.apache.hadoop.fs.Path
{noformat}
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1425:
----------------------------------------------
Status: Patch Available (was: Open)
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830368#action_12830368 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
Yes, the patch works for 10^5 files. The har job took 41mins 21sec.
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1425:
-------------------------------------
Attachment: MAPREDUCE-1425.patch
a patch for trunk.
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828694#action_12828694 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
The archive command is roughly double the memory usage compared with ls. Below are the jmap outputs on 10^5 files
- hadoop archive
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 665091 43671104 [C
2: 1471573 35317752 java.lang.String
3: 261286 20902880 java.net.URI
4: 261276 16721664 org.apache.hadoop.fs.FileStatus
5: 200001 4800024 org.apache.hadoop.fs.permission.FsPermission
6: 261285 4180560 org.apache.hadoop.fs.Path
{noformat}
- hadoop fs -ls
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 304173 21091592 [C
2: 804254 19302096 java.lang.String
3: 100009 8000720 java.net.URI
4: 100001 6400064 org.apache.hadoop.fs.FileStatus
5: 100002 2400048 org.apache.hadoop.fs.permission.FsPermission
6: 100008 1600128 org.apache.hadoop.fs.Path
{noformat}
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1425:
-------------------------------------
Attachment: MAPREDUCE-1425.patch
this patch fixes the problem of too many filestatus objects while creating archvies. Nicholas, can you test this out?
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831756#action_12831756 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
The manual test is simple: run archive on 10^5 files and jmap to read the memory usages as shown previously.
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1425:
-------------------------------------
Attachment: MAPREDUCE-1425_y_0.20.patch
here is a clean patch for y-0.20 branch. I tested the performance issues, and it fixes them. I will upload a patch for the trunk next.
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830211#action_12830211 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
> With the patch + MAPREDUCE-1399 ...
The patch should be applied to y0.20 but not apache 0.20. My fault!
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829842#action_12829842 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
Tried the patch. Got
{noformat}
Exception in archives
null
{noformat}
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831753#action_12831753 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
All tests passed except TestChainErrors, which still failed after the patch had been reverted.
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1425:
-------------------------------------
Attachment: ZOOKEEPER-1425.patch
can you try this patch nicholas? It fixes the issue for me.. can you also try it out?
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, ZOOKEEPER-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831775#action_12831775 ]
Hadoop QA commented on MAPREDUCE-1425:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12435033/MAPREDUCE-1425.patch
against trunk revision 907967.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/console
This message is automatically generated.
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830375#action_12830375 ]
Mahadev konar commented on MAPREDUCE-1425:
------------------------------------------
I dont have a test case here, since we did manual testing to see the memory improvements made from the patch.
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806133#action_12806133 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
Got OutOfMemoryError again after replaced . with full path.
{noformat}
-bash-3.1$ hadoop archive -archiveName t4.har -p /user/tsz t4 .
Exception in thread "LeaseChecker" java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:318)
at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:859)
at org.apache.hadoop.ipc.Client.call(Client.java:719)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.renewLease(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1046)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1058)
at java.lang.Thread.run(Thread.java:619)
10/01/28 22:27:57 WARN hdfs.DFSClient: DataStreamer Exception: java.lang.NullPointerException
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:467)
at org.apache.hadoop.ipc.Client.call(Client.java:720)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
10/01/28 22:27:57 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
10/01/28 22:27:57 WARN hdfs.DFSClient: Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_ljz1el/_har_src_files" - Aborting...
Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_ljz1el/_har_src_files" - Aborting...
{noformat}
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1425:
----------------------------------------------
Status: Open (was: Patch Available)
Hudson is not picking up, re-submitting ...
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830420#action_12830420 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------
After the latest patch, archive uses the same amount of memory as ls. Great job, Mahadev!
- archive
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 304340 21085440 [C
2: 804442 19306608 java.lang.String
3: 100014 8001120 java.net.URI
4: 100001 6400064 org.apache.hadoop.fs.FileStatus
5: 100001 2400024 org.apache.hadoop.fs.permission.FsPermission
{noformat}
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831867#action_12831867 ]
Hudson commented on MAPREDUCE-1425:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #233 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/233/])
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1425:
-------------------------------------
Attachment: (was: ZOOKEEPER-1425.patch)
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1425:
----------------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I have committed this. Thanks, Mahadev!
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425.patch, MAPREDUCE-1425_y_0.20.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1425) archive throws OutOfMemoryError
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-1425:
----------------------------------------------
Attachment: har.sh
m1425_20100129TextFileGenerator.patch
m1425_20100129TextFileGenerator.patch: a program to generate text files.
har.sh: useful commands for testing archive.
> archive throws OutOfMemoryError
> -------------------------------
>
> Key: MAPREDUCE-1425
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: harchive
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Mahadev konar
> Fix For: 0.22.0
>
> Attachments: har.sh, m1425_20100129TextFileGenerator.patch
>
>
> {noformat}
> -bash-3.1$ hadoop archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.regex.Pattern.compile(Pattern.java:1432)
> at java.util.regex.Pattern.<init>(Pattern.java:1133)
> at java.util.regex.Pattern.compile(Pattern.java:847)
> at java.lang.String.replace(String.java:2208)
> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
> at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
> at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
> at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.