You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Mahadev Konar <ma...@hortonworks.com> on 2011/11/01 05:51:52 UTC
Re: Better testing for client code...
I am all for using mockito. I have found it very handy in writing good
unit tests. Our codebase needs little changes to be able to unit test
our codebase.
Writing a "real" unit test for the java client would be a good use
case to see what all changes need to happen on the client.
thanks
mahadev
On Mon, Oct 31, 2011 at 1:20 PM, Patrick Hunt <ph...@apache.org> wrote:
> I've been thinking that we need better testing for the client code. In
> particular it's currently difficult to simulate all the different
> types of failures a client might see. I'm wondering how we might do
> this - two ideas come to mind; 1) aspectj and 2) mocks (mockito).
> Anyone have other suggestions?
>
> I've used 1 before on an ad-hoc basis to find issues in the server
> quorum code (aspects which would randomly cause a connect, bind,
> write, read, etc.... to fail). 2 seems better from a deterministic
> basis (ie repeatable tests that are easier to verify pass/fail). The
> only issue I see with 2 (other than refactoring some of the client
> code) is the need to mock networking level code. Perhaps both
> approaches could be applied together or separately.
>
> Thoughts?
>
> Patrick
>
Re: Better testing for client code...
Posted by Benjamin Reed <br...@apache.org>.
i second the need for such a mocked ZooKeeper client! there are a
couple of failure cases to test in an application that uses ZooKeeper,
but we don't provide a good library to generate them.
ben
On Tue, Nov 1, 2011 at 11:10 AM, Ted Dunning <te...@gmail.com> wrote:
> The testing pain that I have had most is actually the pain of testing a
> client library against failure modes. The scaffolding that would most help
> there would be a mocked Zookeeper client that I could use to trigger
> disconnections and expirations and such. Launching a local server is
> distinctly sub-optimal for several reasons. One is that I can't reliably
> run tests at the same time. Another is that it is hard to trigger possible
> race conditions with an independent server.
>
> It isn't terribly hard to build this, but it has so far exceeded the
> quantum of time that I typically have available.
>
> On Tue, Nov 1, 2011 at 9:11 AM, Patrick Hunt <ph...@apache.org> wrote:
>
>> I'm not suggesting it's an all or nothing proposition. We have a lot
>> of system/functional level testing, which is great. What we lack is
>> unit testing. Take this recent ZOOKEEPER-1271 issue as an example - in
>> many cases it's very hard, at the system level, to exercise all the
>> code paths esp the failure cases. Perhaps if we had more breadth we
>> could have caught this earlier.
>>
>> I'm open to all types of testing, these were just some ideas I had for
>> "big bang for the buck" type scenario. I encourage you all to add
>> additional testing, or new types of testing as you see fit.
>>
>> For example, another area where we are weak is enabling ppl to test
>> their client code, Camille if you contribute that back I think it
>> would be awesome! We could layer other things on top of that, for
>> example some way to trigger different behaviors on the servers which
>> would cause client failover and such. (the problem there is validating
>> correct client behavior).
>>
>> Patrick
>>
>> On Tue, Nov 1, 2011 at 9:01 AM, Fournier, Camille F.
>> <Ca...@gs.com> wrote:
>> > I also like mockito. It has some limitations but generally can be very
>> useful. However, I'm also very much a fan of fixing our code base to be
>> more testable against real servers. I actually like the fact that we use a
>> lot of "integration" testing in our automated build. You do catch things
>> that you would miss in mockito unless you were very good at thinking
>> through all possible orderings of behavior.
>> >
>> > My internal ZK java client testing goes almost entirely against mockito
>> and in retrospect I think I would've been possibly better off using mostly
>> a real server started by the unit tests. Not that I missed a lot of bugs
>> using mockito, but what I gained in speed of tests I lost in readability of
>> the test cases for people that aren't fluent in mocking.
>> >
>> > Aspects are still a deep mystery to me. I know they can be terribly
>> useful but wow are they hard to read.
>> >
>> > I can probably throw together a few quick mockito sample tests to get
>> the ball rolling on this. After I look at some of the patch backlog.
>> >
>> > C
>> >
>> > -----Original Message-----
>> > From: Mahadev Konar [mailto:mahadev@hortonworks.com]
>> > Sent: Tuesday, November 01, 2011 12:52 AM
>> > To: dev@zookeeper.apache.org
>> > Subject: Re: Better testing for client code...
>> >
>> > I am all for using mockito. I have found it very handy in writing good
>> > unit tests. Our codebase needs little changes to be able to unit test
>> > our codebase.
>> >
>> > Writing a "real" unit test for the java client would be a good use
>> > case to see what all changes need to happen on the client.
>> >
>> > thanks
>> > mahadev
>> >
>> > On Mon, Oct 31, 2011 at 1:20 PM, Patrick Hunt <ph...@apache.org> wrote:
>> >> I've been thinking that we need better testing for the client code. In
>> >> particular it's currently difficult to simulate all the different
>> >> types of failures a client might see. I'm wondering how we might do
>> >> this - two ideas come to mind; 1) aspectj and 2) mocks (mockito).
>> >> Anyone have other suggestions?
>> >>
>> >> I've used 1 before on an ad-hoc basis to find issues in the server
>> >> quorum code (aspects which would randomly cause a connect, bind,
>> >> write, read, etc.... to fail). 2 seems better from a deterministic
>> >> basis (ie repeatable tests that are easier to verify pass/fail). The
>> >> only issue I see with 2 (other than refactoring some of the client
>> >> code) is the need to mock networking level code. Perhaps both
>> >> approaches could be applied together or separately.
>> >>
>> >> Thoughts?
>> >>
>> >> Patrick
>> >>
>> >
>>
>
Re: Better testing for client code...
Posted by Ted Dunning <te...@gmail.com>.
The testing pain that I have had most is actually the pain of testing a
client library against failure modes. The scaffolding that would most help
there would be a mocked Zookeeper client that I could use to trigger
disconnections and expirations and such. Launching a local server is
distinctly sub-optimal for several reasons. One is that I can't reliably
run tests at the same time. Another is that it is hard to trigger possible
race conditions with an independent server.
It isn't terribly hard to build this, but it has so far exceeded the
quantum of time that I typically have available.
On Tue, Nov 1, 2011 at 9:11 AM, Patrick Hunt <ph...@apache.org> wrote:
> I'm not suggesting it's an all or nothing proposition. We have a lot
> of system/functional level testing, which is great. What we lack is
> unit testing. Take this recent ZOOKEEPER-1271 issue as an example - in
> many cases it's very hard, at the system level, to exercise all the
> code paths esp the failure cases. Perhaps if we had more breadth we
> could have caught this earlier.
>
> I'm open to all types of testing, these were just some ideas I had for
> "big bang for the buck" type scenario. I encourage you all to add
> additional testing, or new types of testing as you see fit.
>
> For example, another area where we are weak is enabling ppl to test
> their client code, Camille if you contribute that back I think it
> would be awesome! We could layer other things on top of that, for
> example some way to trigger different behaviors on the servers which
> would cause client failover and such. (the problem there is validating
> correct client behavior).
>
> Patrick
>
> On Tue, Nov 1, 2011 at 9:01 AM, Fournier, Camille F.
> <Ca...@gs.com> wrote:
> > I also like mockito. It has some limitations but generally can be very
> useful. However, I'm also very much a fan of fixing our code base to be
> more testable against real servers. I actually like the fact that we use a
> lot of "integration" testing in our automated build. You do catch things
> that you would miss in mockito unless you were very good at thinking
> through all possible orderings of behavior.
> >
> > My internal ZK java client testing goes almost entirely against mockito
> and in retrospect I think I would've been possibly better off using mostly
> a real server started by the unit tests. Not that I missed a lot of bugs
> using mockito, but what I gained in speed of tests I lost in readability of
> the test cases for people that aren't fluent in mocking.
> >
> > Aspects are still a deep mystery to me. I know they can be terribly
> useful but wow are they hard to read.
> >
> > I can probably throw together a few quick mockito sample tests to get
> the ball rolling on this. After I look at some of the patch backlog.
> >
> > C
> >
> > -----Original Message-----
> > From: Mahadev Konar [mailto:mahadev@hortonworks.com]
> > Sent: Tuesday, November 01, 2011 12:52 AM
> > To: dev@zookeeper.apache.org
> > Subject: Re: Better testing for client code...
> >
> > I am all for using mockito. I have found it very handy in writing good
> > unit tests. Our codebase needs little changes to be able to unit test
> > our codebase.
> >
> > Writing a "real" unit test for the java client would be a good use
> > case to see what all changes need to happen on the client.
> >
> > thanks
> > mahadev
> >
> > On Mon, Oct 31, 2011 at 1:20 PM, Patrick Hunt <ph...@apache.org> wrote:
> >> I've been thinking that we need better testing for the client code. In
> >> particular it's currently difficult to simulate all the different
> >> types of failures a client might see. I'm wondering how we might do
> >> this - two ideas come to mind; 1) aspectj and 2) mocks (mockito).
> >> Anyone have other suggestions?
> >>
> >> I've used 1 before on an ad-hoc basis to find issues in the server
> >> quorum code (aspects which would randomly cause a connect, bind,
> >> write, read, etc.... to fail). 2 seems better from a deterministic
> >> basis (ie repeatable tests that are easier to verify pass/fail). The
> >> only issue I see with 2 (other than refactoring some of the client
> >> code) is the need to mock networking level code. Perhaps both
> >> approaches could be applied together or separately.
> >>
> >> Thoughts?
> >>
> >> Patrick
> >>
> >
>
Re: Better testing for client code...
Posted by Patrick Hunt <ph...@apache.org>.
I'm not suggesting it's an all or nothing proposition. We have a lot
of system/functional level testing, which is great. What we lack is
unit testing. Take this recent ZOOKEEPER-1271 issue as an example - in
many cases it's very hard, at the system level, to exercise all the
code paths esp the failure cases. Perhaps if we had more breadth we
could have caught this earlier.
I'm open to all types of testing, these were just some ideas I had for
"big bang for the buck" type scenario. I encourage you all to add
additional testing, or new types of testing as you see fit.
For example, another area where we are weak is enabling ppl to test
their client code, Camille if you contribute that back I think it
would be awesome! We could layer other things on top of that, for
example some way to trigger different behaviors on the servers which
would cause client failover and such. (the problem there is validating
correct client behavior).
Patrick
On Tue, Nov 1, 2011 at 9:01 AM, Fournier, Camille F.
<Ca...@gs.com> wrote:
> I also like mockito. It has some limitations but generally can be very useful. However, I'm also very much a fan of fixing our code base to be more testable against real servers. I actually like the fact that we use a lot of "integration" testing in our automated build. You do catch things that you would miss in mockito unless you were very good at thinking through all possible orderings of behavior.
>
> My internal ZK java client testing goes almost entirely against mockito and in retrospect I think I would've been possibly better off using mostly a real server started by the unit tests. Not that I missed a lot of bugs using mockito, but what I gained in speed of tests I lost in readability of the test cases for people that aren't fluent in mocking.
>
> Aspects are still a deep mystery to me. I know they can be terribly useful but wow are they hard to read.
>
> I can probably throw together a few quick mockito sample tests to get the ball rolling on this. After I look at some of the patch backlog.
>
> C
>
> -----Original Message-----
> From: Mahadev Konar [mailto:mahadev@hortonworks.com]
> Sent: Tuesday, November 01, 2011 12:52 AM
> To: dev@zookeeper.apache.org
> Subject: Re: Better testing for client code...
>
> I am all for using mockito. I have found it very handy in writing good
> unit tests. Our codebase needs little changes to be able to unit test
> our codebase.
>
> Writing a "real" unit test for the java client would be a good use
> case to see what all changes need to happen on the client.
>
> thanks
> mahadev
>
> On Mon, Oct 31, 2011 at 1:20 PM, Patrick Hunt <ph...@apache.org> wrote:
>> I've been thinking that we need better testing for the client code. In
>> particular it's currently difficult to simulate all the different
>> types of failures a client might see. I'm wondering how we might do
>> this - two ideas come to mind; 1) aspectj and 2) mocks (mockito).
>> Anyone have other suggestions?
>>
>> I've used 1 before on an ad-hoc basis to find issues in the server
>> quorum code (aspects which would randomly cause a connect, bind,
>> write, read, etc.... to fail). 2 seems better from a deterministic
>> basis (ie repeatable tests that are easier to verify pass/fail). The
>> only issue I see with 2 (other than refactoring some of the client
>> code) is the need to mock networking level code. Perhaps both
>> approaches could be applied together or separately.
>>
>> Thoughts?
>>
>> Patrick
>>
>
RE: Better testing for client code...
Posted by "Fournier, Camille F." <Ca...@gs.com>.
I also like mockito. It has some limitations but generally can be very useful. However, I'm also very much a fan of fixing our code base to be more testable against real servers. I actually like the fact that we use a lot of "integration" testing in our automated build. You do catch things that you would miss in mockito unless you were very good at thinking through all possible orderings of behavior.
My internal ZK java client testing goes almost entirely against mockito and in retrospect I think I would've been possibly better off using mostly a real server started by the unit tests. Not that I missed a lot of bugs using mockito, but what I gained in speed of tests I lost in readability of the test cases for people that aren't fluent in mocking.
Aspects are still a deep mystery to me. I know they can be terribly useful but wow are they hard to read.
I can probably throw together a few quick mockito sample tests to get the ball rolling on this. After I look at some of the patch backlog.
C
-----Original Message-----
From: Mahadev Konar [mailto:mahadev@hortonworks.com]
Sent: Tuesday, November 01, 2011 12:52 AM
To: dev@zookeeper.apache.org
Subject: Re: Better testing for client code...
I am all for using mockito. I have found it very handy in writing good
unit tests. Our codebase needs little changes to be able to unit test
our codebase.
Writing a "real" unit test for the java client would be a good use
case to see what all changes need to happen on the client.
thanks
mahadev
On Mon, Oct 31, 2011 at 1:20 PM, Patrick Hunt <ph...@apache.org> wrote:
> I've been thinking that we need better testing for the client code. In
> particular it's currently difficult to simulate all the different
> types of failures a client might see. I'm wondering how we might do
> this - two ideas come to mind; 1) aspectj and 2) mocks (mockito).
> Anyone have other suggestions?
>
> I've used 1 before on an ad-hoc basis to find issues in the server
> quorum code (aspects which would randomly cause a connect, bind,
> write, read, etc.... to fail). 2 seems better from a deterministic
> basis (ie repeatable tests that are easier to verify pass/fail). The
> only issue I see with 2 (other than refactoring some of the client
> code) is the need to mock networking level code. Perhaps both
> approaches could be applied together or separately.
>
> Thoughts?
>
> Patrick
>