[jira] Commented: (HADOOP-6248) Circus: Proposal and Preliminary Code for a Hadoop System Testing Framework

["Alex Loddengaard (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753299#action_12753299 ] 

Alex Loddengaard commented on HADOOP-6248:
------------------------------------------

h1. Overview

This proposal defines a system testing framework for Hadoop, named Circus. Circus will let Hadoop testers and users run one or more tests on a Hadoop "context." A context might be a local pseudo-distributed cluster, an EC2 cluster, two Hadoop clusters-each running a different version of Hadoop-or anything else. A test might be the canned pi example, HDFS manipulations, an HDFS upgrade, custom MapReduce code that is compiled and checked for validity, a distcp across two different versions of Hadoop, or anything else. This proposal will provide more examples of contexts and jobs in its later sections.

h1. System Testing vs. Unit Testing

One might wonder how Circus is different from a unit testing framework such as JUnit.  A unit testing framework is meant to run short-lived tests, one after another. Unit tests, by definition, are meant to test small, independent pieces of a larger entity.  A system test, in general, is larger than a unit test, in that a system test might run for several hours, analyze logs, run on a large cluster of machines, or test the runtime and performance of a job. We will see more use cases later.

h1. Motivation and Use Cases

# Hadoop users want to know if their MapReduce jobs are compatible with new Hadoop releases.  They should have a tool that lets them easily spin up a cluster running a particular version of Hadoop and compile and run MapReduce jobs on that cluster.  Ideally users will put their MR jobs in Circus and put Circus in their QA pipeline.  Then, when considering an upgrade, they can make minimal changes to test those same jobs in a newer release of Hadoop.
# Hadoop users want to know how their MapReduce jobs will perform in a new Hadoop release.
# Hadoop QA engineers want to run complicated tests such as HDFS upgrades, distcp jobs between two different versions of HDFS, etc.  Testers want these tests to all be part of the same framework so a suite of regression tests can be run with a single command.
# Hadoop QA engineers want to continually be running Hadoop jobs to ensure API compatibility between versions.  Circus could power a community-driven website, where community members upload Hadoop jobs and sample data that help generate reports on backwards compatibility, both at the API level and at the runtime level.

h1. Expected Usage

I imagine that large users of Hadoop can port their MapReduce jobs to work in this framework, and add Circus to their QA pipeline.  They might also configure the local context to work with their dev cluster.  I imagine Hadoop QA engineers can use Circus to test some of the tricky use cases of Hadoop, such as cross-version distcp jobs, HDFS upgrades, or anything else.  Testing these use cases will require that the QA engineer write contexts and tests.

h1. Implementation Details

Please see the source code.  A README file exists that explains interfaces and other implementation details.

h1. Open Issues and Questions

# Can this framework be useful to Hadoop developers?  And if so, how?
## Should Circus provide hooks for inducing failure in a Hadoop cluster?
## Might a trunk context be useful so developers can spin up a cluster very quickly from their local SVN checkout?
## Can the local context allow the user to specify a core jar to be used, instead of a release?
# What other canned contexts would be useful besides the local context?  Probably an EC2 context.
# How can this framework cater to performance testing?  Is just providing time information sufficient?
# Should tests specify the contexts they are supposed to run in?  Seems like tests should at least have the option to.  This will be particularly interesting for tests that need to run on two Hadoop clusters.
# Should the environment variables specified by a context be validated?  That is, should the interface for contexts be more rigid?
# Should Circus provide a facility for analyzing the Hadoop logs generated during the run of a test?

> Circus: Proposal and Preliminary Code for a Hadoop System Testing Framework
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6248
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>         Environment: Python, bash
>            Reporter: Alex Loddengaard
>
> This issue contains a proposal and preliminary source code for Circus, a Hadoop system testing framework.  At a high level, Circus will help Hadoop users and QA engineers to run system tests on a configurable Hadoop cluster, or distribution of Hadoop.  See the comment below for the proposal itself.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.