You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org> on 2010/04/19 13:17:50 UTC
[jira] Created: (AVRO-521) Out of memory and other issues with
Junit tests for mapreduce
Out of memory and other issues with Junit tests for mapreduce
-------------------------------------------------------------
Key: AVRO-521
URL: https://issues.apache.org/jira/browse/AVRO-521
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.4.0
Reporter: Thiruvalluvan M. G.
Assignee: Thiruvalluvan M. G.
The forthcoming patch fixed the out of memory problem we encounter while running JUnit tests for mapreduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-521) Out of memory and other issues with
Junit tests for mapreduce
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-521:
-------------------------------------
Status: Patch Available (was: Open)
> Out of memory and other issues with Junit tests for mapreduce
> -------------------------------------------------------------
>
> Key: AVRO-521
> URL: https://issues.apache.org/jira/browse/AVRO-521
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.4.0
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-521.patch
>
>
> The forthcoming patch fixed the out of memory problem we encounter while running JUnit tests for mapreduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-521) Out of memory and other issues with
Junit tests for mapreduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859526#action_12859526 ]
Doug Cutting commented on AVRO-521:
-----------------------------------
Thanks for updating these tests. I started with a Hadoop JUnit3-style test and neglected to update them.
We should suppress these deprecation warnings. This uses the Hadoop 0.20 APIs, which, while deprecated in 0.20, have now since been declared the long-term stable Hadoop APIs. There are new APIs, but they're not yet stable, and also are not present in older releases. I believe this code should work in 0.18 and 0.19, and the forthcoming 0.21, but have not tested these yet.
> Out of memory and other issues with Junit tests for mapreduce
> -------------------------------------------------------------
>
> Key: AVRO-521
> URL: https://issues.apache.org/jira/browse/AVRO-521
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.4.0
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-521.patch
>
>
> The forthcoming patch fixed the out of memory problem we encounter while running JUnit tests for mapreduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-521) Out of memory and other issues with
Junit tests for mapreduce
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-521:
-------------------------------------
Attachment: AVRO-521.patch
Increasing the max memory to 128 MB (from the default 64 MB) fixes the out of memory problem.
But there is one more issue. TestWordCountGeneric leaves behind the (empty) output directory and the following test TestWordCountSpecific fails to run complaining that the output directory already exists.
I fixed that problem by removing the output directory after completing each of these tests. Since I was at it, I moved the JUnit tests toJUnit4 style from JUnit3 style. All other tests are in JUnit4. Also added a couple of @Override annotations.
There are quite a few warnings because we use mapreduce's JobConf. What is the new way to submit jobs? The documentation says we should use Configuration. I'm not sure how to do that. Can anybody help us with correct usage? Thanks.
> Out of memory and other issues with Junit tests for mapreduce
> -------------------------------------------------------------
>
> Key: AVRO-521
> URL: https://issues.apache.org/jira/browse/AVRO-521
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.4.0
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-521.patch
>
>
> The forthcoming patch fixed the out of memory problem we encounter while running JUnit tests for mapreduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-521) Out of memory and other issues with
Junit tests for mapreduce
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-521:
-------------------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
Committed revision 937307 after adding @SuppressWarnings annotation for JobConf deprecation. Thanks Doug for the help.
> Out of memory and other issues with Junit tests for mapreduce
> -------------------------------------------------------------
>
> Key: AVRO-521
> URL: https://issues.apache.org/jira/browse/AVRO-521
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.4.0
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-521.patch
>
>
> The forthcoming patch fixed the out of memory problem we encounter while running JUnit tests for mapreduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.