You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "George Porter (JIRA)" <ji...@apache.org> on 2009/02/04 03:59:59 UTC
[jira] Created: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
------------------------------------------------------------------------------------
Key: HADOOP-5164
URL: https://issues.apache.org/jira/browse/HADOOP-5164
Project: Hadoop Core
Issue Type: Bug
Components: test
Reporter: George Porter
Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "George Porter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Porter updated HADOOP-5164:
----------------------------------
Status: Patch Available (was: Open)
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.1.patch, HADOOP-5164.2.patch, HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "George Porter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Porter updated HADOOP-5164:
----------------------------------
Attachment: HADOOP-5164.1.patch
With this patch, every test case that is in a subclass of ClusterMapReduceTestCase now explicitly calls startCluster(). This makes it possible to create test case subclasses of ClusterMapReduceTestCase that use alternative parameters to the cluster. It also makes it possible to have different clusters in your test cases that have different parameters.
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.1.patch, HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "George Porter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Porter updated HADOOP-5164:
----------------------------------
Hadoop Flags: [Incompatible change]
Modified this issue to be an "incompatible change"
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.1.patch, HADOOP-5164.2.patch, HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "George Porter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674434#action_12674434 ]
George Porter commented on HADOOP-5164:
---------------------------------------
Thanks for the feedback Vinod.
I went ahead and refactored out all of the startCluster() calls and made each test case explicitly call them, rather than having an implicit call to startCluster(true, null) in setUp(). This should both solve my original problem, as well as give the test writer finer-grained control over the parameters passed to each test cluster. What do you think?
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.1.patch, HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "George Porter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670223#action_12670223 ]
George Porter commented on HADOOP-5164:
---------------------------------------
I recommend that we create a protected method called getDefaultClusterProperties() that can be overridden by subclasses of ClusterMapReduceTestCase.
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "George Porter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Porter updated HADOOP-5164:
----------------------------------
Attachment: HADOOP-5164.patch
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "George Porter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Porter updated HADOOP-5164:
----------------------------------
Attachment: HADOOP-5164.2.patch
This updated patch includes an additional test that needed to be modified to directly call setupCluster(true, null).
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.1.patch, HADOOP-5164.2.patch, HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670313#action_12670313 ]
Vinod K V commented on HADOOP-5164:
-----------------------------------
Even with this patch, we cannot have different cluster properties in different tests of a single test-case. This, as well as the general problem being pointed, seems to have the only solution of overriding setUp() to do nothing and calling startCluster explicitly in each test with any required config properties.
Further, I don't clearly understand why you want to be able to invoke TestCase.setUp(), it does nothing anyways.
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675093#action_12675093 ]
Hadoop QA commented on HADOOP-5164:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12400457/HADOOP-5164.2.patch
against trunk revision 745705.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 27 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3882/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3882/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3882/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3882/console
This message is automatically generated.
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.1.patch, HADOOP-5164.2.patch, HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5164:
----------------------------------
Status: Open (was: Patch Available)
For a cluster shared between tests, wouldn't the correct idiom use TestSetup, as in TestAppend3, TestReduceFetch, etc.? If each test needs a different set of parameters, it's not clear to me why one wouldn't simply create a Configuration with these parameters, and start Mini\*Cluster with it...
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.1.patch, HADOOP-5164.2.patch, HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5164) Subclasses of
ClusterMapReduceTestCase can't easily add new configuration parameters
Posted by "George Porter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Porter updated HADOOP-5164:
----------------------------------
Release Note: Modified the behavior of ClusterMapReduceTestCase so that setupCluster() is no longer called automatically in TestCase.setUp(). Now, you must explicitly call setupCluster() for each test you wish to run (or you can share a single cluster among several tests).
Added a release note.
> Subclasses of ClusterMapReduceTestCase can't easily add new configuration parameters
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5164
> URL: https://issues.apache.org/jira/browse/HADOOP-5164
> Project: Hadoop Core
> Issue Type: Bug
> Components: test
> Reporter: George Porter
> Attachments: HADOOP-5164.1.patch, HADOOP-5164.2.patch, HADOOP-5164.patch
>
>
> Currently there is not a clean way for subclasses of ClusterMapReduceTestCase to add to the JobConf used to start the cluster daemons.
> The startCluster() method does take a Properties object that is added to the JobConf used to the start the daemons. However, startCluster() is called from JUnit inside the setUp() method, which sets this parameter to be null.
> If you try to override setUp() in a subclass of ClusterMapReduceTestCase, then you won't be able to invoke the TestCase.setUp() ancestor without calling ClusterMapReduceTestCase's setUp() (which will pass in the null parameter). On the other hand, if you just call startCluster() within your test method, then you would be starting up a cluster that was already started.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.