You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Nigel Daley <nd...@yahoo-inc.com> on 2008/11/20 07:23:48 UTC
Feature Designs and Test Plans
Hadoop Committers,
The Hadoop project has grown to become a critical platform for many
companies. As we approach a 1.0 release (perhaps within the next 6
months) I think it is time we adopt some more rigor around designing
and testing new features.
As committers, we are already given a couple duties that relate to
quality. From http://wiki.apache.org/hadoop/HowToCommit :
> If a committer reviews a patch they've not authored, and believe it
> to be of sufficient quality, then they can commit the patch,
> otherwise the patch should be cancelled with a clear explanation for
> why it was rejected.
>
> ...
>
> Patches should be rejected which do not adhere to the guidelines in http://wiki.apache.org/hadoop/HowToContribute
> and to the http://wiki.apache.org/hadoop/CodeReviewChecklist.
> Committers should always be polite to contributors and try to
> instruct and encourage them to contribute better patches.
>
These have been a good starting point, but I'd like to propose more
rigor.
I propose that before we commit issues marked as "New Feature", they
must have:
1. a design doc attachment
2. a test plan attachment
(Templates to be provided for both)
I believe these 2 items will enhance the communication, design, and
quality of our new features.
Are committers open to adopting these further guidelines?
Thanks,
Nige
Re: Feature Designs and Test Plans
Posted by Nigel Daley <nd...@yahoo-inc.com>.
Thanks Peter, but code review guidelines are a different discussion.
I'm asking for commitment to feature design docs and test plans in
this thread.
Nige
On Nov 19, 2008, at 10:47 PM, Peter Veentjer wrote:
> What about thread safety?
>
> I see that the code base is very large and it is quite hard to analyse
> because there is not much documentation if classes are used by
> multiple threads/single thread/immutable
> Other things I see are:
> - eating up interrupted exceptions
> - very coarse grained locking
> - locking on objects you don't own and nested locking (both deadlock
> prone)
> - using non threadsafe collections in combination with custom locking
> (doesn't need to be a bad thing, but not by default)
> - java memory model problems (and they are hard to analyse because of
> the lack of documentation)
>
> These are a few of the things I have found so far.
>
> I think these things can be simplified by adding them to the
> guidelines.
>
> On Thu, Nov 20, 2008 at 7:23 AM, Nigel Daley <nd...@yahoo-inc.com>
> wrote:
>> Hadoop Committers,
>>
>> The Hadoop project has grown to become a critical platform for many
>> companies. As we approach a 1.0 release (perhaps within the next 6
>> months)
>> I think it is time we adopt some more rigor around designing and
>> testing new
>> features.
>>
>> As committers, we are already given a couple duties that relate to
>> quality.
>> From http://wiki.apache.org/hadoop/HowToCommit :
>>
>>> If a committer reviews a patch they've not authored, and believe
>>> it to be
>>> of sufficient quality, then they can commit the patch, otherwise
>>> the patch
>>> should be cancelled with a clear explanation for why it was
>>> rejected.
>>>
>>> ...
>>>
>>> Patches should be rejected which do not adhere to the guidelines in
>>> http://wiki.apache.org/hadoop/HowToContribute and to the
>>> http://wiki.apache.org/hadoop/CodeReviewChecklist. Committers
>>> should always
>>> be polite to contributors and try to instruct and encourage them to
>>> contribute better patches.
>>>
>> These have been a good starting point, but I'd like to propose more
>> rigor.
>>
>> I propose that before we commit issues marked as "New Feature",
>> they must
>> have:
>> 1. a design doc attachment
>> 2. a test plan attachment
>> (Templates to be provided for both)
>>
>> I believe these 2 items will enhance the communication, design, and
>> quality
>> of our new features.
>>
>> Are committers open to adopting these further guidelines?
>>
>> Thanks,
>> Nige
>>
>>
Re: Feature Designs and Test Plans
Posted by Peter Veentjer <al...@gmail.com>.
What about thread safety?
I see that the code base is very large and it is quite hard to analyse
because there is not much documentation if classes are used by
multiple threads/single thread/immutable
Other things I see are:
- eating up interrupted exceptions
- very coarse grained locking
- locking on objects you don't own and nested locking (both deadlock prone)
- using non threadsafe collections in combination with custom locking
(doesn't need to be a bad thing, but not by default)
- java memory model problems (and they are hard to analyse because of
the lack of documentation)
These are a few of the things I have found so far.
I think these things can be simplified by adding them to the guidelines.
On Thu, Nov 20, 2008 at 7:23 AM, Nigel Daley <nd...@yahoo-inc.com> wrote:
> Hadoop Committers,
>
> The Hadoop project has grown to become a critical platform for many
> companies. As we approach a 1.0 release (perhaps within the next 6 months)
> I think it is time we adopt some more rigor around designing and testing new
> features.
>
> As committers, we are already given a couple duties that relate to quality.
> From http://wiki.apache.org/hadoop/HowToCommit :
>
>> If a committer reviews a patch they've not authored, and believe it to be
>> of sufficient quality, then they can commit the patch, otherwise the patch
>> should be cancelled with a clear explanation for why it was rejected.
>>
>> ...
>>
>> Patches should be rejected which do not adhere to the guidelines in
>> http://wiki.apache.org/hadoop/HowToContribute and to the
>> http://wiki.apache.org/hadoop/CodeReviewChecklist. Committers should always
>> be polite to contributors and try to instruct and encourage them to
>> contribute better patches.
>>
> These have been a good starting point, but I'd like to propose more rigor.
>
> I propose that before we commit issues marked as "New Feature", they must
> have:
> 1. a design doc attachment
> 2. a test plan attachment
> (Templates to be provided for both)
>
> I believe these 2 items will enhance the communication, design, and quality
> of our new features.
>
> Are committers open to adopting these further guidelines?
>
> Thanks,
> Nige
>
>
Re: Feature Designs and Test Plans
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Nov 19, 2008, at 10:23 PM, Nigel Daley wrote:
> I propose that before we commit issues marked as "New Feature", they
> must have:
> 1. a design doc attachment
> 2. a test plan attachment
> (Templates to be provided for both)
>
> I believe these 2 items will enhance the communication, design, and
> quality of our new features.
>
> Are committers open to adopting these further guidelines?
>
+1 with the expectation that common sense will prevail for smaller
features. (Maybe we should have a jira-type for "Big Feature" ? *smile*)
Nigel, do you have a proposal for the templates?
Maybe it would be good to have a simple textual template with the
happy implication that people can put the design up as a jira comment
(with markup?) after the discussion (comments) along with an attachment.
Implication: it is easier for a late gate-crasher to ignore
discussions and concentrate on the the 'proposal' as it evolves over
time.
Arun
Re: Feature Designs and Test Plans
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
+1 to require design docs and test plans for new features.
--Konstantin
Nigel Daley wrote:
> Hadoop Committers,
>
> The Hadoop project has grown to become a critical platform for many
> companies. As we approach a 1.0 release (perhaps within the next 6
> months) I think it is time we adopt some more rigor around designing and
> testing new features.
>
> As committers, we are already given a couple duties that relate to
> quality. From http://wiki.apache.org/hadoop/HowToCommit :
>
>> If a committer reviews a patch they've not authored, and believe it to
>> be of sufficient quality, then they can commit the patch, otherwise
>> the patch should be cancelled with a clear explanation for why it was
>> rejected.
>>
>> ...
>>
>> Patches should be rejected which do not adhere to the guidelines in
>> http://wiki.apache.org/hadoop/HowToContribute and to the
>> http://wiki.apache.org/hadoop/CodeReviewChecklist. Committers should
>> always be polite to contributors and try to instruct and encourage
>> them to contribute better patches.
>>
> These have been a good starting point, but I'd like to propose more rigor.
>
> I propose that before we commit issues marked as "New Feature", they
> must have:
> 1. a design doc attachment
> 2. a test plan attachment
> (Templates to be provided for both)
>
> I believe these 2 items will enhance the communication, design, and
> quality of our new features.
>
> Are committers open to adopting these further guidelines?
>
> Thanks,
> Nige
>
>
Re: Feature Designs and Test Plans
Posted by Dhruba Borthakur <dh...@gmail.com>.
+1.
I would write a design doc and a test plan according to the above guidelines
for an upcoming new feature "symbolic links" via HADOOP-4044.
thanks,
dhruba
On Tue, Feb 17, 2009 at 11:02 PM, Nigel Daley <nd...@yahoo-inc.com> wrote:
> [let's see if this formatting comes through better...]
>
>
> [resurrecting an old thread now that I've had time to put together a
> proposal]
>
> Sounds like folks are agreeing we need design docs and test plans for
> features, with common sense prevailing on when to apply this rule to smaller
> features.
>
> So, let me propose some design doc and test plan templates.
>
> Design Doc Template:
> • Problem Definition
> • what problem does the feature solve?
> • why is it needed?
> • Use Cases
> • sequence of simple steps on how users interacts with the
> feature
> • Interaction with Current System
> • does this feature build on an existing feature? if so,
> how?
> • how does the user interact with the new feature from
> outside the system
> • how does the new feature interact within the system
> • Requirements
> • focus on what functionality must exist and minimize
> discussion of how that functionality will be implemented
> • list items out of scope
> • Design
> • give a high level description of the implementation
> •
> algorithms and/or data-structures involved, this may be in the form of
> pseudo-code
> • failure analysis
> • Future Work
> • what additional work is related or could be done, but is
> not included in this design
>
> Test Plan Template:
> • Feature(s) Tested: enumerate the feature(s)
> • which Jira issue(s)?
> • what is the feature?
> • what is the externally visible view of the feature?
> • Risk Scenarios: enumerate the bad things that could happen in the
> system that either:
> • could be caused by the feature
> • could have an effect on the feature
> • Test Cases: enumerate all tests in tables
> • using table headings Id, Type of Test, Description,
> Expected Behavior, Is Automated?
> • enumerate positive and negative test case types
> • include functional, system, scale, and performance test
> case types
> • indicate which tests are automated in JUnit suite
>
> Past Design Doc Examples
> •
> http://issues.apache.org/jira/secure/attachment/12348296/DFSUpgradeProposal3.html
>
> Past Test Plan Examples
> •
> https://issues.apache.org/jira/secure/attachment/12373559/PermissionsTestPlan2.pdf
> •
> https://issues.apache.org/jira/secure/attachment/12363605/BlockCrcFeatureTestPlan.pdf
> •
> https://issues.apache.org/jira/secure/attachment/12351299/TestPlan-HdfsUpgrade.html
>
> Some examples requiring the above:
> • HADOOP-1700 (file append)
> • HADOOP-702 (DFS upgrades)
> • HADOOP-3938 (disk quotas)
> • HADOOP-372 (input format per dir)
> • HADOOP-153 (skip bad records)
>
>
> Thoughts? Is this something we can all commit to do?
>
> Cheers,
> Nige
>
>
> Hadoop Committers,
>>
>> The Hadoop project has grown to become a critical platform for many
>> companies. As we approach a 1.0 release (perhaps within the next 6 months)
>> I think it is time we adopt some more rigor around designing and testing new
>> features.
>>
>> As committers, we are already given a couple duties that relate to
>> quality. Fromhttp://wiki.apache.org/hadoop/HowToCommit :
>>
>> If a committer reviews a patch they've not authored, and believe it to be
>>> of sufficient quality, then they can commit the patch, otherwise the patch
>>> should be cancelled with a clear explanation for why it was rejected.
>>>
>>> ...
>>>
>>> Patches should be rejected which do not adhere to the guidelines
>>> inhttp://wiki.apache.org/hadoop/HowToContribute and to thehttp://
>>> wiki.apache.org/hadoop/CodeReviewChecklist. Committers should always be
>>> polite to contributors and try to instruct and encourage them to contribute
>>> better patches.
>>>
>>> These have been a good starting point, but I'd like to propose more
>> rigor.
>>
>> I propose that before we commit issues marked as "New Feature", they must
>> have:
>> 1. a design doc attachment
>> 2. a test plan attachment
>> (Templates to be provided for both)
>>
>> I believe these 2 items will enhance the communication, design, and
>> quality of our new features.
>>
>> Are committers open to adopting these further guidelines?
>>
>> Thanks,
>> Nige
>>
>
>
Re: Feature Designs and Test Plans
Posted by Nigel Daley <nd...@yahoo-inc.com>.
[let's see if this formatting comes through better...]
[resurrecting an old thread now that I've had time to put together a
proposal]
Sounds like folks are agreeing we need design docs and test plans for
features, with common sense prevailing on when to apply this rule to
smaller features.
So, let me propose some design doc and test plan templates.
Design Doc Template:
• Problem Definition
• what problem does the feature solve?
• why is it needed?
• Use Cases
• sequence of simple steps on how users interacts with the feature
• Interaction with Current System
• does this feature build on an existing feature? if so, how?
• how does the user interact with the new feature from outside the
system
• how does the new feature interact within the system
• Requirements
• focus on what functionality must exist and minimize discussion of
how that functionality will be implemented
• list items out of scope
• Design
• give a high level description of the implementation
•
algorithms and/or data-structures involved, this may be in the form of
pseudo-code
• failure analysis
• Future Work
• what additional work is related or could be done, but is not
included in this design
Test Plan Template:
• Feature(s) Tested: enumerate the feature(s)
• which Jira issue(s)?
• what is the feature?
• what is the externally visible view of the feature?
• Risk Scenarios: enumerate the bad things that could happen in the
system that either:
• could be caused by the feature
• could have an effect on the feature
• Test Cases: enumerate all tests in tables
• using table headings Id, Type of Test, Description, Expected
Behavior, Is Automated?
• enumerate positive and negative test case types
• include functional, system, scale, and performance test case types
• indicate which tests are automated in JUnit suite
Past Design Doc Examples
• http://issues.apache.org/jira/secure/attachment/12348296/DFSUpgradeProposal3.html
Past Test Plan Examples
• https://issues.apache.org/jira/secure/attachment/12373559/PermissionsTestPlan2.pdf
• https://issues.apache.org/jira/secure/attachment/12363605/BlockCrcFeatureTestPlan.pdf
• https://issues.apache.org/jira/secure/attachment/12351299/TestPlan-HdfsUpgrade.html
Some examples requiring the above:
• HADOOP-1700 (file append)
• HADOOP-702 (DFS upgrades)
• HADOOP-3938 (disk quotas)
• HADOOP-372 (input format per dir)
• HADOOP-153 (skip bad records)
Thoughts? Is this something we can all commit to do?
Cheers,
Nige
> Hadoop Committers,
>
> The Hadoop project has grown to become a critical platform for many
> companies. As we approach a 1.0 release (perhaps within the next 6
> months) I think it is time we adopt some more rigor around designing
> and testing new features.
>
> As committers, we are already given a couple duties that relate to
> quality. Fromhttp://wiki.apache.org/hadoop/HowToCommit :
>
>> If a committer reviews a patch they've not authored, and believe it
>> to be of sufficient quality, then they can commit the patch,
>> otherwise the patch should be cancelled with a clear explanation
>> for why it was rejected.
>>
>> ...
>>
>> Patches should be rejected which do not adhere to the guidelines inhttp://wiki.apache.org/hadoop/HowToContribute
>> and to thehttp://wiki.apache.org/hadoop/CodeReviewChecklist.
>> Committers should always be polite to contributors and try to
>> instruct and encourage them to contribute better patches.
>>
> These have been a good starting point, but I'd like to propose more
> rigor.
>
> I propose that before we commit issues marked as "New Feature", they
> must have:
> 1. a design doc attachment
> 2. a test plan attachment
> (Templates to be provided for both)
>
> I believe these 2 items will enhance the communication, design, and
> quality of our new features.
>
> Are committers open to adopting these further guidelines?
>
> Thanks,
> Nige
Re: Feature Designs and Test Plans
Posted by Nigel Daley <nd...@yahoo-inc.com>.
[resurrecting an old thread now that I've had time to put together a
proposal]
Sounds like folks are agreeing we need design docs and test plans for
features, with common sense prevailing on when to apply this rule to
smaller features.
So, let me propose some design doc and test plan templates.
Design Doc Template:
Problem Definition
what problem does the feature solve?
why is it needed?
Use Cases
sequence of simple steps on how users interacts with the feature
Interaction with Current System
does this feature build on an existing feature? if so, how?
how does the user interact with the new feature from outside the system
how does the new feature interact within the system
Requirements
focus on what functionality must exist and minimize discussion of how
that functionality will be implemented
list items out of scope
Design
give a high level description of the implementation
algorithms and/or data-structures involved, this may be in the form of
pseudo-code
failure analysis
Future Work
what additional work is related or could be done, but is not included
in this design
Test Plan Template:
Feature(s) Tested: enumerate the feature(s)
which Jira issue(s)?
what is the feature?
what is the externally visible view of the feature?
Risk Scenarios: enumerate the bad things that could happen in the
system that either:
could be caused by the feature
could have an effect on the feature
Test Cases: enumerate all tests in tables
using table headings Id, Type of Test, Description, Expected Behavior,
Is Automated?
enumerate positive and negative test case types
include functional, system, scale, and performance test case types
indicate which tests are automated in JUnit suite
Past Design Doc Examples
http://issues.apache.org/jira/secure/attachment/12348296/DFSUpgradeProposal3.html
Past Test Plan Examples
https://issues.apache.org/jira/secure/attachment/12373559/PermissionsTestPlan2.pdf
https://issues.apache.org/jira/secure/attachment/12363605/BlockCrcFeatureTestPlan.pdf
https://issues.apache.org/jira/secure/attachment/12351299/TestPlan-HdfsUpgrade.html
Some examples requiring the above:
HADOOP-1700 (file append)
HADOOP-702 (DFS upgrades)
HADOOP-3938 (disk quotas)
HADOOP-372 (input format per dir)
HADOOP-153 (skip bad records)
Thoughts? Is this something we can all commit to do?
Cheers,
Nige
> Hadoop Committers,
>
> The Hadoop project has grown to become a critical platform for many
> companies. As we approach a 1.0 release (perhaps within the next 6
> months) I think it is time we adopt some more rigor around designing
> and testing new features.
>
> As committers, we are already given a couple duties that relate to
> quality. Fromhttp://wiki.apache.org/hadoop/HowToCommit :
>
>> If a committer reviews a patch they've not authored, and believe it
>> to be of sufficient quality, then they can commit the patch,
>> otherwise the patch should be cancelled with a clear explanation
>> for why it was rejected.
>>
>> ...
>>
>> Patches should be rejected which do not adhere to the guidelines inhttp://wiki.apache.org/hadoop/HowToContribute
>> and to thehttp://wiki.apache.org/hadoop/CodeReviewChecklist.
>> Committers should always be polite to contributors and try to
>> instruct and encourage them to contribute better patches.
>>
> These have been a good starting point, but I'd like to propose more
> rigor.
>
> I propose that before we commit issues marked as "New Feature", they
> must have:
> 1. a design doc attachment
> 2. a test plan attachment
> (Templates to be provided for both)
>
> I believe these 2 items will enhance the communication, design, and
> quality of our new features.
>
> Are committers open to adopting these further guidelines?
>
> Thanks,
> Nige
Re: Feature Designs and Test Plans
Posted by Nigel Daley <nd...@yahoo-inc.com>.
On Nov 20, 2008, at 10:54 AM, Doug Cutting wrote:
> Nigel Daley wrote:
>> I propose that before we commit issues marked as "New Feature",
>> they must have:
>> 1. a design doc attachment
>> 2. a test plan attachment
>> (Templates to be provided for both)
>
> Clarifying test expectations is a good idea in principle. I gather
> that what you're after in (1) is an enumeration of new functionality
Yes, how the new feature interacts within the system and how to
interact with it from outside the system (if applicable).
> and in (2) corresponding tests. That sounds reasonable for large
> new features, but might be overkill for small features, no?
I agree with Arun: "+1 with the expectation that common sense will
prevail for smaller features". I'll come up with some examples.
> I'd like to see the templates and some examples. Documents would
> preferably be in some open format, like HTML or plain text.
Ya, I'll work on these next if we have agreement in principle.
> Is there an assumption that, before the patch is committed, that the
> tests in the plan will be implemented and passed?
Yes, for the automatable tests.
> Are these to be automated tests, with the goal that we'll end up
> with automated regression tests for all features?
I think all tests should be enumerated in the test plan (positive and
negative functional, system, scale, performance, etc) with the
expectation that the positive and negative functional tests will be
automated with JUnit.
Nige
Re: Feature Designs and Test Plans
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
> Is there an assumption that, before the patch is committed, that the
> tests in the plan will be implemented and passed?
> Are these to be
> automated tests, with the goal that we'll end up with automated
> regression tests for all features?
1. I think we should have a requirement to have all tests that could be
automated be automated and implemented as JUnit tests.
Some tests cannot be automated and may require manual testing or testing
on a large scale, which developers may not have access to.
2. Therefore, we may have to commit before the entire test plan is completed.
But the plan should be developed.
Konstantin
Doug Cutting wrote:
> Nigel Daley wrote:
>> I propose that before we commit issues marked as "New Feature", they
>> must have:
>> 1. a design doc attachment
>> 2. a test plan attachment
>> (Templates to be provided for both)
>
> Clarifying test expectations is a good idea in principle. I gather that
> what you're after in (1) is an enumeration of new functionality and in
> (2) corresponding tests. That sounds reasonable for large new features,
> but might be overkill for small features, no?
>
> I'd like to see the templates and some examples. Documents would
> preferably be in some open format, like HTML or plain text.
>
> Is there an assumption that, before the patch is committed, that the
> tests in the plan will be implemented and passed? Are these to be
> automated tests, with the goal that we'll end up with automated
> regression tests for all features?
>
> Doug
>
>
>
Re: Feature Designs and Test Plans
Posted by Doug Cutting <cu...@apache.org>.
Nigel Daley wrote:
> I propose that before we commit issues marked as "New Feature", they
> must have:
> 1. a design doc attachment
> 2. a test plan attachment
> (Templates to be provided for both)
Clarifying test expectations is a good idea in principle. I gather that
what you're after in (1) is an enumeration of new functionality and in
(2) corresponding tests. That sounds reasonable for large new features,
but might be overkill for small features, no?
I'd like to see the templates and some examples. Documents would
preferably be in some open format, like HTML or plain text.
Is there an assumption that, before the patch is committed, that the
tests in the plan will be implemented and passed? Are these to be
automated tests, with the goal that we'll end up with automated
regression tests for all features?
Doug