You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2011/10/11 20:55:11 UTC
[jira] [Created] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
---------------------------------------------------------------------------------------
Key: MAPREDUCE-3169
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1, mrv2, test
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138734#comment-13138734 ]
Todd Lipcon commented on MAPREDUCE-3169:
----------------------------------------
I don't think we can feasibly update MiniMRCluster, since its constructors have lots of MR1-isms. If we break the constructor APIs, then we can't have upstream projects (eg HBase) compile against 20 and 23 interchangeably.
Just to make sure we're on the same page, here are the requirements as I see it:
1. Upstream projects must be able to start/stop a mini-cluster suitable for testing MR jobs in a close-to-real-life environment.
2. When building against 0.20 they should start MR1 clusters, when building against 0.23 they should start YARN/MR2 clusters.
3. Upstream projects should not be expected to move to any API that isn't supported by 0.20 (lots of people will continue to run 0.20 for at least a year if not longer)
4. Existing MR1 tests that don't depend on MR1 specifics should be updated to run against MR2 in trunk/23.
Are we all agreed on the above?
The open questions are:
1. Should we repurpose the existing classes or create a new class?
2. Do we need to do changes to 0.20.x series to create a compatible API?
I'm not sure exactly what you mean by "Re-implement old MiniMRCluster to be a facade on MiniMRClientCluster keeping only meaningful methods" -- but it seems like you're suggesting API changes on MiniMRCluster in trunk. That would break requirement #3 above.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152299#comment-13152299 ]
Arun C Murthy commented on MAPREDUCE-3169:
------------------------------------------
Should we call it something other than MiniMRCluster to prevent confusion?
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152820#comment-13152820 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #80 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/80/])
Merge -r 1203370:1203371 from trunk to branch. FIXES: MAPREDUCE-3169
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203402
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.20.205.1, 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137564#comment-13137564 ]
Tom White commented on MAPREDUCE-3169:
--------------------------------------
Right, but in the migration path I'm suggesting, if they run against 0.20 they use the jobtracker implementation of MiniMRCluster, and if they run against 0.23 they use the YARN implementation. There are some MR1-isms in those constructors (e.g. jobTrackerPort) that we would keep, but nothing that I can see that would cause a problem. Does that make sense?
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161167#comment-13161167 ]
Siddharth Seth commented on MAPREDUCE-3169:
-------------------------------------------
Committed the amendment to deprecate MiniMRCluster to trunk and branch-0.23. Thanks Ahmend
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3169:
------------------------------------
Attachment: MAPREDUCE-3169-0.20-security_rev2.patch
The updated version of the patch for branch 0.20-security (incorporating Tom's comments).
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152846#comment-13152846 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #97 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/97/])
Merge -r 1203370:1203371 from trunk to branch. FIXES: MAPREDUCE-3169
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203402
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.20.205.1, 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3169:
------------------------------------
Target Version/s: 0.20.206.0, 0.23.0 (was: 0.23.0, 0.20.206.0)
Assignee: Ahmed Radwan
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138654#comment-13138654 ]
Alejandro Abdelnur commented on MAPREDUCE-3169:
-----------------------------------------------
How about the following approach?
* Don't do anything pre 0.23
* For 0.23/trunk:
** Have new MiniMRClientCluster (h-mr-p/h-mr-c/h-mr-c-jc)
** Re-implement old MiniMRCluster to be a facade on MiniMRClientCluster keeping only meaningful methods (h-mr-p/h-mr-regression-test)
** Use MiniMRCluster for all (70 odd) testcases that use it for regression purposes (the current 'ant tests'). For the rest (20 odd) testcases that use MR1 specific stuff, we just delete those tests.
** MiniMRCluster would in h-mr-p/h-mr-regression-test module, other projects (pig, hive, oozie, etc) would use it from there. It would not be avail in MR2 modules (to force all new stuff to use the new MiniMRClientCluster)
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141310#comment-13141310 ]
Tom White commented on MAPREDUCE-3169:
--------------------------------------
Another nice benefit of this approach (either of Todd's 1 or 2) is that we could write a LocalJobRunner implementation of MiniMRCluster and get increased test coverage (e.g. by writing a parameterized test superclass, see http://junit.sourceforge.net/javadoc/org/junit/runners/Parameterized.html). This would be a follow up JIRA.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151497#comment-13151497 ]
Alejandro Abdelnur commented on MAPREDUCE-3169:
-----------------------------------------------
+1 (I've been using without a hitch it to get streaming testcases to work as part of HADOOP-7590)
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149465#comment-13149465 ]
Hadoop QA commented on MAPREDUCE-3169:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503565/MAPREDUCE-3169-0.20-security_rev2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1299//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1299//console
This message is automatically generated.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139000#comment-13139000 ]
Alejandro Abdelnur commented on MAPREDUCE-3169:
-----------------------------------------------
Got the point why we need the new class in 0.20, it make sense.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152315#comment-13152315 ]
Tom White commented on MAPREDUCE-3169:
--------------------------------------
> Should we call it something other than MiniMRCluster to prevent confusion?
Yes. The patch for 0.20 does not change MiniMRCluster, and for 23/trunk it provides an implementation of MiniMRCluster for the operations that are possible when running with YARN (the jobtracker/tasktracker-specific methods will throw UnsupportedOperationException). In addition, a new interface called MiniMRClientCluster is introduced (in 20/23/trunk) that downstream projects can start using.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145802#comment-13145802 ]
Ahmed Radwan commented on MAPREDUCE-3169:
-----------------------------------------
For organization purpose, please note that the actual changes made to specific tests for the purpose of migration are tacked in MAPREDUCE-3369. This current ticket is only tracking general interfaces changes.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161159#comment-13161159 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1426 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1426/])
MAPREDUCE-3169 amendment. Deprecate MiniMRCluster. Contributed by Ahmed Radwan)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209235
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152831#comment-13152831 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #867 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/867/])
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203371
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.20.205.1, 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152361#comment-13152361 ]
Siddharth Seth commented on MAPREDUCE-3169:
-------------------------------------------
As part of this, should MiniMRCluster be deprecated in 23/trunk in favour of MiniMRClientClusterFactory, or an additional note in MiniMRCluster pointing people to MiniMRClientClusterFactory.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152309#comment-13152309 ]
Ahmed Radwan commented on MAPREDUCE-3169:
-----------------------------------------
I agree, this is how old tests can work with almost none or very minor changes. Thanks Arun & Tucu!
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161609#comment-13161609 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #113 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/113/])
merge MAPREDUCE-3169 amendment (Deprecate MiniMRCluster) from trunk. Constributed by Ahmed Radwan
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209239
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137557#comment-13137557 ]
Todd Lipcon commented on MAPREDUCE-3169:
----------------------------------------
The issue is that HBase, Pig, and Hive obtain MiniMRClusters directly from a constructor, rather than any kind of factory method. So I think renaming it is impossible without causing a breaking change for all of those -- even if they only use the "client operations".
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146416#comment-13146416 ]
Tom White commented on MAPREDUCE-3169:
--------------------------------------
Looks good to me.
* Why do some operations in MiniMRCluster do nothing, while others throw a UnsupportedOperationException?
* Please add class-level javadoc.
* MiniMRClientClusterFactory should use Class<?>, not a raw Class.
* MiniMRYarnClusterAdapter might be better named MiniMRClientYarnCluster.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152340#comment-13152340 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1353 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1353/])
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203371
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3169:
-------------------------------------
Target Version/s: 0.20.206.0, 0.23.0 (was: 0.23.0, 0.20.206.0)
Status: Open (was: Patch Available)
Good discussion, I'll spend a bit more time on this soon. A bit swamped this week.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3169:
------------------------------------
Attachment: MAPREDUCE-3169-truck.patch
MAPREDUCE-3169-0.20-security.patch
I am attaching two patches; one for trunk and the other for 0.20-security.
I have added a new test case for both patches. It is a basic MR test for checking job submission and basic counters. The test classes in both patches are identical, but it'll use MiniMRYarnCluster on truck and MiniMRCluster on 0.20-security.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152422#comment-13152422 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #177 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/177/])
Merge -r 1203370:1203371 from trunk to branch. FIXES: MAPREDUCE-3169
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203402
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152862#comment-13152862 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #901 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/901/])
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203371
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.20.205.1, 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161281#comment-13161281 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1378 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1378/])
MAPREDUCE-3369. Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169. Contributed by Ahmed Radwan.
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209281
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/HadoopTestCase.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileOutputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalMRNotification.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTaskCommit.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/UtilsForTests.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/WordCount.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/JobControlTestUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapReduceLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestChainErrors.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestMapReduceChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestSingleElementChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMultipleInputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/map
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/map/TestMultithreadedMapper.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestMRMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestMRKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobCounters.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestLocalMRNotification.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestTaskCommit.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestMapReduceLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestChainErrors.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestMapReduceChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestSingleElementChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMultipleInputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/map/TestMultithreadedMapper.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestMRMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/partition/TestMRKeyFieldBasedComparator.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3169:
------------------------------------
Target Version/s: 0.20.206.0, 0.23.0 (was: 0.23.0, 0.20.206.0)
Status: Patch Available (was: Open)
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152665#comment-13152665 ]
Tom White commented on MAPREDUCE-3169:
--------------------------------------
> should MiniMRCluster be deprecated in 23/trunk in favour of MiniMRClientClusterFactory
Yes, I think that's a good idea.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.20.205.1, 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161296#comment-13161296 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1354 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1354/])
MAPREDUCE-3369. Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169. Contributed by Ahmed Radwan.
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209281
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/HadoopTestCase.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileOutputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalMRNotification.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTaskCommit.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/UtilsForTests.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/WordCount.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/JobControlTestUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapReduceLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestChainErrors.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestMapReduceChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestSingleElementChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMultipleInputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/map
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/map/TestMultithreadedMapper.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestMRMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestMRKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobCounters.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestLocalMRNotification.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestTaskCommit.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestMapReduceLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestChainErrors.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestMapReduceChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestSingleElementChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMultipleInputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/map/TestMultithreadedMapper.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestMRMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/partition/TestMRKeyFieldBasedComparator.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152369#comment-13152369 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1303 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1303/])
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203371
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137582#comment-13137582 ]
Todd Lipcon commented on MAPREDUCE-3169:
----------------------------------------
Yea, all of the MR-isms in the constructor is what worries me. I don't really want to maintain an API long-term with JT details, etc. Making this API a blessed part of 0.23 just means we have to hold on to it longer, no?
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137493#comment-13137493 ]
Tom White commented on MAPREDUCE-3169:
--------------------------------------
Looks good.
There are lots of tests that use MiniMRCluster - these need to be ported to use the new interface and moved into the mvn tree. Those which test JobTracker and TaskTracker internals shouldn't be updated, of course, but there are many client-only tests that serve as regression tests for MR which we need to be running. I'd like to see this done as a part of this work since going through this exercise will likely inform the MiniMRClientCluster API.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161160#comment-13161160 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1352 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1352/])
MAPREDUCE-3169 amendment. Deprecate MiniMRCluster. Contributed by Ahmed Radwan)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209235
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152440#comment-13152440 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #189 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/189/])
Merge -r 1203370:1203371 from trunk to branch. FIXES: MAPREDUCE-3169
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203402
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161207#comment-13161207 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1376 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1376/])
MAPREDUCE-3169 amendment. Deprecate MiniMRCluster. Contributed by Ahmed Radwan)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209235
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141402#comment-13141402 ]
Ahmed Radwan commented on MAPREDUCE-3169:
-----------------------------------------
Yes, this will be a very nice benefit.
I am now going through the tests changing them to try to assess which way is better.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137536#comment-13137536 ]
Ahmed Radwan commented on MAPREDUCE-3169:
-----------------------------------------
Sure, I wanted first to get feedback on this core change, before going forward and changing a lot of dependent tests (mainly client-side, input/output formats, etc.) that can be updated to use the new interface.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161630#comment-13161630 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #915 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/915/])
MAPREDUCE-3369. Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169. Contributed by Ahmed Radwan.
MAPREDUCE-3169 amendment. Deprecate MiniMRCluster. Contributed by Ahmed Radwan)
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209281
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/HadoopTestCase.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileOutputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalMRNotification.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTaskCommit.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/UtilsForTests.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/WordCount.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/JobControlTestUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapReduceLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestChainErrors.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestMapReduceChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestSingleElementChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMultipleInputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/map
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/map/TestMultithreadedMapper.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestMRMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestMRKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobCounters.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestLocalMRNotification.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestTaskCommit.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestMapReduceLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestChainErrors.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestMapReduceChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestSingleElementChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMultipleInputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/map/TestMultithreadedMapper.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestMRMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/partition/TestMRKeyFieldBasedComparator.java
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209235
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3169:
------------------------------------
Attachment: MAPREDUCE-3169-trunk_rev3.patch
Thanks Tom!
Please see updated patch incorporating your comments.
> Why do some operations in MiniMRCluster do nothing, while others throw a UnsupportedOperationException?
This was a subjective assessment, so methods that are tightly coupled with MR1 will throw an UnsupportedOperationException so they'll noisily fail. Other methods are just unimplemented as the corresponding functionality is abscent in MR2. I can change all of them to throw an UnsupportedOperationException but wanted to be as flexible as possible with old code.
> Please add class-level javadoc.
Added to all new classes.
> MiniMRClientClusterFactory should use Class<?>, not a raw Class.
Change incorporated in the new patch.
> MiniMRYarnClusterAdapter might be better named MiniMRClientYarnCluster.
The name used is "MiniMRClientCluster". The MiniMRYarnClusterAdapter name is used by the adapter for MiniMRYarnCluster providing the MiniMRClientCluster interface.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3169:
------------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.1
0.24.0
0.20.205.1
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk, 0.23 & 0.20.205 branches.
Thanks Ahmed.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.20.205.1, 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137610#comment-13137610 ]
Tom White commented on MAPREDUCE-3169:
--------------------------------------
We could go either route - the original proposal just means we need to change 100+ tests in Hadoop (in at least two branches) at the same time. I just checked and HBase, Pig and Hive use MiniMRCluster in only a few places, so the change there is manageable.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161178#comment-13161178 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #243 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/243/])
merge MAPREDUCE-3169 amendment (Deprecate MiniMRCluster) from trunk. Constributed by Ahmed Radwan
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209239
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152421#comment-13152421 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #176 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/176/])
Merge -r 1203370:1203371 from trunk to branch. FIXES: MAPREDUCE-3169
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203402
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152305#comment-13152305 ]
Alejandro Abdelnur commented on MAPREDUCE-3169:
-----------------------------------------------
MiniMRCluster is on purpose to allow downstream projects to continue using the same API for testing for all their legacy testcases with none/minimal disruption.
For new stuff folks should use MiniMRClientCluster via MiniMRClientClusterFactory.
I'm planning to commit this now, are we OK?
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161580#comment-13161580 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #95 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/95/])
merge MAPREDUCE-3169 amendment (Deprecate MiniMRCluster) from trunk. Constributed by Ahmed Radwan
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209239
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161288#comment-13161288 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1428 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1428/])
MAPREDUCE-3369. Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169. Contributed by Ahmed Radwan.
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209281
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/HadoopTestCase.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileOutputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalMRNotification.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTaskCommit.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/UtilsForTests.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/WordCount.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/JobControlTestUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapReduceLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestChainErrors.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestMapReduceChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestSingleElementChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMultipleInputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/map
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/map/TestMultithreadedMapper.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestMRMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestMRKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobCounters.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestLocalMRNotification.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestTaskCommit.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestMapReduceLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestChainErrors.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestMapReduceChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestSingleElementChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMultipleInputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/map/TestMultithreadedMapper.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestMRMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/partition/TestMRKeyFieldBasedComparator.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161176#comment-13161176 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #240 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/240/])
merge MAPREDUCE-3169 amendment (Deprecate MiniMRCluster) from trunk. Constributed by Ahmed Radwan
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209239
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Matt Foley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated MAPREDUCE-3169:
----------------------------------
Target Version/s: 0.23.1, 1.0.0 (was: 0.23.0, 1.1.0)
Fix Version/s: (was: 0.24.0)
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138812#comment-13138812 ]
Alejandro Abdelnur commented on MAPREDUCE-3169:
-----------------------------------------------
I'm suggesting that MiniMRCluster becomes a wrapper of MiniMRClientCluster. The constructors would be kept not break old tests, only propagating configs that make sense to MiniMRClientCluster.
In addition MiniMRCluster API should be either trimmed of MR1 specific methods or make them throw UnsupportedOperationException.
While this would break some tests, most likely will be MR1 tests only. And, if it is the case for other projects, then those projects will have to make some changes to accomodate MR2. (although I'd believe this would be minimal or NIL).
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3169:
------------------------------------
Target Version/s: 0.20.206.0, 0.23.0 (was: 0.23.0, 0.20.206.0)
Status: Patch Available (was: Open)
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146000#comment-13146000 ]
Hadoop QA commented on MAPREDUCE-3169:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502804/MAPREDUCE-3169-trunk_rev2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1265//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1265//console
This message is automatically generated.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161217#comment-13161217 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #253 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/253/])
merge MAPREDUCE-3169 amendment (Deprecate MiniMRCluster) from trunk. Constributed by Ahmed Radwan
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209239
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149462#comment-13149462 ]
Hadoop QA commented on MAPREDUCE-3169:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12503564/MAPREDUCE-3169-trunk_rev3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1298//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1298//console
This message is automatically generated.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137545#comment-13137545 ]
Tom White commented on MAPREDUCE-3169:
--------------------------------------
I was thinking about a way of implementing this which is less invasive to tests that use MiniMRCluster by keeping its client interface the same.
In 0.20 (and 0.22) we would have a MiniMR1Cluster class that is a copy of the existing MiniMRCluster, then we would remove the non-client-side methods from MiniMRCluster. (Test code is not a part of the published API so it is acceptable to change it incompatibly, and in fact this change would minimize the amount of changes needed by tests.) The few tests that use the framework-specific features of MiniMRCluster would be changed to use MiniMR1Cluster, the others would not need changing.
In 0.23 and trunk we would have the same cutdown MiniMRCluster and the tests that use it would be moved to the mvn tree.
HBase, Pig, and Hive would not need to change unless they use non-client-side methods (I don't know to what extent they do).
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137491#comment-13137491 ]
Hadoop QA commented on MAPREDUCE-3169:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501146/MAPREDUCE-3169-truck.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 170 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed the unit tests build
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1176//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1176//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1176//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1176//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1176//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1176//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-web-proxy.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1176//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1176//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1176//console
This message is automatically generated.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152338#comment-13152338 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1279 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1279/])
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2. (Ahmed via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1203371
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRYarnClusterAdapter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3169:
------------------------------------
Attachment: MAPREDUCE-3169-trunk_deprecation_amendment.patch
> should MiniMRCluster be deprecated in 23/trunk in favour of MiniMRClientClusterFactory.
Agree, attaching a small amendment to deprecate MiniMRCluster on trunk, so it is clear to use MiniMRClientClusterFactory instead.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.20.205.1, 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3169:
------------------------------------
Attachment: MAPREDUCE-3169-trunk_rev2.patch
A revised patch for trunk. The patch mainly adds a mock class MiniMRCluster that wraps the new interfaces, so old tests can run with minimal changes. There are also some minor changes to the patch based on issues faced during tests migration.
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161587#comment-13161587 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Hdfs-HAbranch-build #3 (See [https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/3/])
merge MAPREDUCE-3169 amendment (Deprecate MiniMRCluster) from trunk. Constributed by Ahmed Radwan
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209239
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster
equivalent which only provides client APIs cross MR1 and MR2
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161596#comment-13161596 ]
Hudson commented on MAPREDUCE-3169:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #882 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/882/])
MAPREDUCE-3369. Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169. Contributed by Ahmed Radwan.
MAPREDUCE-3169 amendment. Deprecate MiniMRCluster. Contributed by Ahmed Radwan)
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209281
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/HadoopTestCase.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileOutputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalMRNotification.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTaskCommit.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/UtilsForTests.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/WordCount.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/JobControlTestUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapReduceLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestChainErrors.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestMapReduceChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/chain/TestSingleElementChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMultipleInputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/map
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/map/TestMultithreadedMapper.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestMRMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestMRKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobCounters.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestLocalMRNotification.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSpecialCharactersInOutputPath.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestTaskCommit.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestKeyFieldBasedComparator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestChild.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestMapReduceLazyOutput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestChainErrors.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestMapReduceChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/chain/TestSingleElementChain.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestMultipleInputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/jobcontrol/TestMapReduceJobControl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/map/TestMultithreadedMapper.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestMRMultipleOutputs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/partition/TestMRKeyFieldBasedComparator.java
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1209235
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
> Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, mrv2, test
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ahmed Radwan
> Fix For: 0.23.1, 1.0.0
>
> Attachments: MAPREDUCE-3169-0.20-security.patch, MAPREDUCE-3169-0.20-security_rev2.patch, MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_deprecation_amendment.patch, MAPREDUCE-3169-trunk_rev2.patch, MAPREDUCE-3169-trunk_rev3.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible" with MiniMRCluster for most applications, and the MR1 implementation could be backported to 20x branch. Thus, dependent projects like HBase could migrate to this implementation and test against both MR1 and MR2. We can also use this to port over the current functional tests that use only the client-side features of MiniMRCluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira