You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2009/10/25 07:47:59 UTC
[jira] Created: (MAPREDUCE-1154) Large-scale, automated test
framwork for Map-Reduce
Large-scale, automated test framwork for Map-Reduce
---------------------------------------------------
Key: MAPREDUCE-1154
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: test
Reporter: Arun C Murthy
Fix For: 0.21.0
HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop.
This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1154) Large-scale, automated test
framwork for Map-Reduce
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769759#action_12769759 ]
Arun C Murthy commented on MAPREDUCE-1154:
------------------------------------------
Some Map-Reduce specific functionality to provide a flavour for the proposal:
{noformat}
/**
* Daemon/job artifacts.
*/
public enum Artifact {
LOGS,
OUTPUTS
}
/**
* Get tasktrackers on the given rack.
* @param cluster map-reduce <code>Cluster</code>
* @param rackId rack
* @return tasktrackers on the given rack
* @throws IOException
* @throws InterruptedException
*/
public static List<TaskTracker> getTaskTrackersOnRack(Cluster cluster,
String rackId)
throws IOException, InterruptedException;
/**
* Get tasks running on the given tasktracker.
* @param cluster map-reduce <code>Cluster</code>
* @param taskTracker task-tracker
* @return tasks running on the given tasktracker
* @throws IOException
* @throws InterruptedException
*/
public static List<TaskAttemptID>
getTasksRunningOnTaskTracker(Cluster cluster, TaskTracker taskTracker)
throws IOException, InterruptedException;
/**
* Get tasktracker on the given host.
* @param cluster map-reduce <code>Cluster</code>
* @param hostName host
* @return tasktracker on the given host
* @throws IOException
* @throws InterruptedException
*/
public static TaskTracker getTaskTracker(Cluster cluster, String hostName)
throws IOException, InterruptedException;
/**
* Get tasktrackers on which tasks of the given job executed.
* @param job map-reduce <code>Job</code>
* @param taskAttemptIDs optional list of task-attempts whose tasktrackers
* need to be fetched
* @return tasktrackers on which tasks of the given job executed, optionally
* for a specific set of task-attempts
* @throws IOException
* @throws InterruptedException
*/
public static List<TaskTracker>
getTaskTrackersForJob(Job job, TaskAttemptID ... taskAttemptIDs)
throws IOException, InterruptedException;
/**
* Fetch artifacts of a given job (optionally a specific set of task-attempts of
* the job).
* @param job job whose logs are to be fetched
* @param type artifact type
* @param dir directory to place the fetched logs
* @param taskAttemptIDs optional list of task-attempts of the job
* @throws IOException
*/
public static void fetchJobArtifacts(Job job,
Artifact type, Path dir,
TaskAttemptID ... taskAttemptIDs)
throws IOException;
/**
* Fetch job artifacts and check if they have the <code>pattern</code>.
* @param job map-reduce job
* @param pattern pattern to check
* @param fetch if <code>true</code> fetch the artifacts into
* <code>dir</code>, else do not fetch
* @param dir directory to place the fetched outputs
* @param taskAttempts optional tasks of the job
* @return <code>true</code> if the artifacts contain <code>pattern</code>,
* <code>false</code> otherwise
* @throws IOException
*/
public static boolean checkJobArtifacts(Job job,
Artifact type, String pattern,
boolean fetch, Path dir,
TaskAttemptID... taskAttempts)
throws IOException;
/**
* Kill the given task-tracker
* @param cluster map-reduce <code>Cluster</code>
* @param taskTracker <code>TaskTracker</code> to be killed
* @throws IOException
* @throws InterruptedException
*/
public static void killTaskTracker(Cluster cluster, TaskTracker taskTracker)
throws IOException, InterruptedException;
/**
* Kill a given task-attempt running on the given tasktracker
* @param cluster map-reduce <code>Cluster</code>
* @param taskTracker <code>TaskTracker</code> on which the task is running
* @param taskAttemptId task-attempt to be killed
* @throws IOException
* @throws InterruptedException
*/
public static void killTask(Cluster cluster, TaskTracker taskTracker,
TaskAttemptID taskAttemptId)
throws IOException, InterruptedException;
{noformat}
> Large-scale, automated test framwork for Map-Reduce
> ---------------------------------------------------
>
> Key: MAPREDUCE-1154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: test
> Reporter: Arun C Murthy
> Fix For: 0.21.0
>
>
> HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop.
> This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1154) Large-scale, automated test
framwork for Map-Reduce
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874424#action_12874424 ]
Konstantin Boudnik commented on MAPREDUCE-1154:
-----------------------------------------------
I'd suggest to close this JIRA because it has been largely covered by HADOOP-6332 and coming soon MAPREDUCE-1774
> Large-scale, automated test framwork for Map-Reduce
> ---------------------------------------------------
>
> Key: MAPREDUCE-1154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: test
> Reporter: Arun C Murthy
> Fix For: 0.21.0
>
> Attachments: testing.patch
>
>
> HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop.
> This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1154) Large-scale, automated test
framwork for Map-Reduce
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769898#action_12769898 ]
Philip Zeyliger commented on MAPREDUCE-1154:
--------------------------------------------
I'm vaguely uncomfortable with having a lot of code, even though it's test code, weaved in via AspectJ. It seems like it will make it very easy to make changes that break the testing code (because the testing code is not visible to the regular tools, and is in an unexpected place). I understand, of course, that the build system will check that the weaving can happen, but since these tests are inherently large-scale and not run at every Hudson (or are they?), it worries me a bit.
Has anyone done the "reverse" and "unweaved" functions from classes? Seems like we could annotate functions with @RemoveInProduction, and then use some tool to forcibly remove methods from the resulting .class files. Still opaque, but at least it's clear where the testing code is.
If you've already been working with this in AspectJ, I'm curious how the experience has been.
-- Philip
> Large-scale, automated test framwork for Map-Reduce
> ---------------------------------------------------
>
> Key: MAPREDUCE-1154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: test
> Reporter: Arun C Murthy
> Fix For: 0.21.0
>
> Attachments: testing.patch
>
>
> HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop.
> This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1154) Large-scale, automated test
framwork for Map-Reduce
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787039#action_12787039 ]
Konstantin Boudnik commented on MAPREDUCE-1154:
-----------------------------------------------
bq. Has anyone done the "reverse" and "unweaved" functions from classes?
One way to do this is to create {{around}} advice for all calls of the method you are interested in. This advice will do nothing but return immediately.
> Large-scale, automated test framwork for Map-Reduce
> ---------------------------------------------------
>
> Key: MAPREDUCE-1154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: test
> Reporter: Arun C Murthy
> Fix For: 0.21.0
>
> Attachments: testing.patch
>
>
> HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop.
> This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1154) Large-scale, automated test
framwork for Map-Reduce
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Boudnik resolved MAPREDUCE-1154.
-------------------------------------------
Resolution: Duplicate
This has been addressed as HADOOP-6332 and derived work.
> Large-scale, automated test framwork for Map-Reduce
> ---------------------------------------------------
>
> Key: MAPREDUCE-1154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: test
> Reporter: Arun C Murthy
> Attachments: testing.patch
>
>
> HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop.
> This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1154) Large-scale, automated test
framwork for Map-Reduce
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-1154:
-------------------------------------
Attachment: testing.patch
The attached patch has the skeleton of:
# Proposed utilities for Map-Reduce
# Newer apis we will need in the Map-Reduce framework to support the utilites:
# enhancements to Job/Cluster for enhanced inspect-ability & control-ability
# functionality to fetch logs for daemons/tasks etc.,
# a new 'DataLoader' interface to provide input-data for large-scale map-reduce test-jobs. DataLoader has 3 basic implementations: DataGenerator (e.g. randomwriter), DistcpDataLoader, LocalDataLoader (fs -copyFromLocal).
# A sample test (ClusterTest.java) which exhibits how these things fit together.
PS: The changes to Job.java and Cluster.java are for the purposes of illustration only, eventually we will use aspects to weave them in for test-builds. We do not want these interfaces to be available on production clusters...
> Large-scale, automated test framwork for Map-Reduce
> ---------------------------------------------------
>
> Key: MAPREDUCE-1154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: test
> Reporter: Arun C Murthy
> Fix For: 0.21.0
>
> Attachments: testing.patch
>
>
> HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop.
> This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1154) Large-scale, automated test
framwork for Map-Reduce
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769762#action_12769762 ]
Arun C Murthy commented on MAPREDUCE-1154:
------------------------------------------
ClusterTest.java:
{noformat}
public class ClusterTest extends TestCase {
Cluster cluster;
Configuration conf;
@Override
protected void setUp() throws Exception {
conf = new Configuration();
ClusterTestUtils.setupCluster(conf);
cluster = new Cluster(conf);
}
@Test
public void testCluster() throws Exception {
// Load data
DataLoader dataLoader = new DistcpDataLoader();
List<Path> inputs = new ArrayList<Path>();
inputs.add(new Path("in"));
dataLoader.load(inputs, new Path("out"));
// Setup job
Job job = Job.getInstance(cluster);
job.setJobName("cluster-test-mr-job");
// Run job
long start = System.currentTimeMillis();
job.submit();
job.waitForCompletion(true);
long end = System.currentTimeMillis();
FileContext fc = FileContext.getLocalFSFileContext();
// Fetch daemon logs and validate them
Path daemonLogs = new Path("daemon-logs");
assertFalse("JobTracker logs contain FATAL warnings!",
MRClusterTestUtils.checkDaemonLogs(
cluster, LogSource.JOBTRACKER, start, end,
"FATAL", true, daemonLogs) == false);
assertFalse("TaskTracker logs contain FATAL warnings!",
MRClusterTestUtils.checkDaemonLogs(
cluster, LogSource.TASKTRACKER, start, end,
"FATAL", true, daemonLogs) == false);
fc.delete(daemonLogs, true);
// Fetch job logs, outputs etc. and validate them
Path jobLogs = new Path("daemon-logs");
Path jobOutputs = new Path("daemon-logs");
TaskAttemptID m0 = null; // map 0
TaskAttemptID m1 = null; // map 1
TaskAttemptID r0 = null; // reduce 0
assertFalse("Task log contains FATAL warnings!",
MRClusterTestUtils.checkJobArtifacts(job,
Artifact.LOGS, "FATAL",
true, jobLogs) == false);
assertFalse("Task log contains WARN warnings!",
MRClusterTestUtils.checkJobArtifacts(
job, Artifact.LOGS, "WARN",
false, jobLogs, m0, m1) == false);
assertTrue("Task outputs do not contain expected output!",
MRClusterTestUtils.checkJobArtifacts(
job, Artifact.OUTPUTS,
"hello, world!", true, jobOutputs, r0));
fc.delete(jobLogs, true);
fc.delete(jobOutputs, true);
}
@Override
protected void tearDown() throws Exception {
ClusterTestUtils.tearDownCluster(conf);
}
}
{noformat}
> Large-scale, automated test framwork for Map-Reduce
> ---------------------------------------------------
>
> Key: MAPREDUCE-1154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: test
> Reporter: Arun C Murthy
> Fix For: 0.21.0
>
> Attachments: testing.patch
>
>
> HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop.
> This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.