Mocking framework

[Jesse Yates <je...@gmail.com>]

Hi all,

Currently Accumulo doesn't have any mocking framework. However, in some
recent work (ACCUMULO-53) doing good testing was greatly facilitated by
adding in a mocking framework. They make it easy to only test the piece of
the architecture that we are interested in testing, helping to isolate and
ensure that we are really fixing problems in the code.

So, I would like to propose adding the following test-time dependencies for
testing:

EasyMock (http://easymock.org/)
This is a really popular mocking framework for java and is really easy to
use. I've rarely run into a situation that wasn't covered with what EasyMock
does.

PowerMock (http://code.google.com/p/powermock/)
This is the utility to cover what other mocking frameworks (including
Mockito) can't do - capturing creation of objects (new Whatever(...)) and
returning mocks instead, mocking static calls, mocking standard java lib
calls, and accessing private methods/fields.

This combination has proven to be incredibly effective and easy to use in
the past, allowing for a really comprehensive testing suite.

 If you want to see an example of this stuff in action, check out the patch
'java_ACCUMULO-53_v2.patch' on ACCUMULO-53.

What does everyone think?

Thanks!
--Jesse

-------------------
Jesse Yates
240-888-2200
@jesse_yates

Re: Mocking framework

[Jesse Yates <je...@gmail.com>]

On Fri, Oct 28, 2011 at 8:19 AM, John W Vines <jo...@ugov.gov> wrote:

> I will start off by saying that myself, as well as most of the committers,
> are no familiar with EasyMock, PowerMock, or really any Mocking tools. But
> I've done some reading to determine what the scope of mocking can get us. So
> pardon me if I bore you because I'm just going to lay out what I know in
> hopes of A. making sure my understanding is correct and B. to see if what I
> think we should use it for is within the scope of Mocking as well as
> feasible.
>
> 1. With exception for PowerMock, mocking tools are nothing more than tools
> which give the ability to define a pre-set set of responses to an interface.
> That is, if we wanted to implement a low level iterator to replace the
> Reader we use to read items off disk/out of the in memory map, we could, but
> we could only give it a set of responses to give when queried. We could not
> (through just the mocking tools) set it up to give programmatic responses,
> such as returning the result of another method, unless we 'pre-load' the
> method result. That is, we couldn't have one mock method which put an item
> in a map and another mock method pull it out again, unless we define the
> pull method to return that specific value for the Xth call to the pull
> method to which that correlated.
>

That isn't exactly correct. You can actually pass in the data structure so
when someone does a call (e.g. put(Thing)) you can catch that call and and
actually pass that into a data structure which is accessed by reference. For
example, check out EasyMock's IAnswer where you can programatically
determine a 'return' result.



>
> 2. PowerMock lets us essentially do the same as things like EasyMock,
> except we're not bound to interfaces. This way we can use a defined class
> and only overwrite the methods we want with predefined results.
>
> 3. What I want to see us doing, at a very high level, is to have the
> ability to mock an entire TServer to the extent where we will use something
> to replace Zookeeper (We should probably turn our ZK work with an interface)
> with a MockZookeeper (not generated through a Mock util) which is nothing
> more than a Map. Same thing with the FileReader, except a SortedMap, the
> loggers, and the master. This way we could fully implement a whole TServer
> without worry about HDFS and Zookeeper. To a similar extent I would like to
> see this done for all core components, but mocking the various connectors we
> use to get done what we need to. I see a few sets of Mock class we will have
> to create. But with less chance of divergence in behavior then we currently
> experience with our MockAccumulo setup.
>


I feel like this kind of thing (particularly with a fully mocked TServer)
would be part of
Accumulo-14<https://issues.apache.org/jira/browse/ACCUMULO-14>and the
testing suite. Otherwise, you end up with divergent implementations
and have the same mess where the mocks don't have any real correlation to
how the system works.

 However, a lot of times you really don't want to (and shouldn't be) mocking
the full interface for a component (eg. a full mock TSever) since then you
are not really testing much of anything other than you are making the calls
to the mock that you can see you are making from the code (I'm thinking here
particularly of code that basically does pass through of calls to other
objects). In that case you really need to have an integration test to make
sure that each piece works with others, since they would be clearly tightly
coupled.

The mock is really optimal for cases where you have loose coupling between
objects and only need to cover a few calls to an external class. You are
really looking to test the logic/computation of the object and only cover a
few elements of interaction with other objects via mocking.


> 4. My principal concern about 3 above is divergence with the code (less so
> than our current setup but with emulating thrift interfaces we could get
> problems). But also the feasibility of mocking thrift connections. But if
> all this works out, I think we'll be able to optimize our code coverage
> better than our current setup without the requirement to do a full Accumulo
> setup for junit tests.
>
>
This is already the problem with the MockAccumulo (and everything under
client.impl.mock) so we want to avoid re-redoing that - divergent behavior
is only bad :)

In short, mocking is really powerful and great for doing isolated testing,
but in the end is only as good as the people who use it. It is very easy to
go overboard with the mocking and basically mock out an entire, tightly
coupled interface that has no reflection on reality. This leads to people
thinking their code works because it matches the mock responses, which
actually look nothing like what reality would serve up to their code (one
case where I have been bitten by this in the past if mocking out what I
_thought_ ZooKeeper would return).



> Cheers
> John
>
> ----- Original Message -----
> | From: "Jesse Yates" <je...@gmail.com>
> | To: accumulo-dev@incubator.apache.org
> | Sent: Thursday, October 27, 2011 2:21:33 PM
> | Subject: Re: Mocking framework
> | On Thu, Oct 27, 2011 at 10:55 AM, Drew Farris <dr...@apache.org> wrote:
> |
> | > I agree in principle without having reviewed the tests accompanying
> | > ACCUMULO-53.
> | >
> | > I've used EasyMock in the past and have been very happy with the
> | > ability to programatically mock implementations of interfaces,
> | > define
> | > test input and assert a proper sequence of calls, and output. In
> | > general I think mock object testing is a useful tool, especially in
> | > cases where the existing interface implementations have heavyweight
> | > external dependencies, such as for mocking JDBC result sets or
> | > portions of the Hadoop framework. I've used mock objects to write
> | > unit
> | > tests for mappers and reducers by mocking out the hadoop context,
> | > counters, etc. Arguably this would be have been better achieved with
> | > MRUnit, which wasn't on my radar at the time.
> | >
> | > I found that I didn't have to jump through hoops to make code
> | > testable. Private methods could really be private instead of
> | > protected
> | > or public in order to facilitate testing.
> |
> |
> | The comment about testing private methods was only a comment on the
> | power of
> | the framework, not on what should be done. IMHO you shouldn't be
> | testing
> | private methods w/o a very good reason.
> |
> |
> | > I also appreciate the
> | > ability to not only assert return values, but assert a series of
> | > method calls and side effects.
> | >
> | > The danger with this is only testing in isolation using mocks -- If
> | > the mocks don't follow the contract or conventions the
> | > implementations
> | > the tests are useless or deceptive. Implementation class behavior
> | > can
> | > change in ways that are invisible from an api perspective, so it's
> | > important that these sorts of changes be reflected adequately by the
> | > mocks themselves.
> |
> |
> | Of course there is also the case where you end up just checking to
> | make sure
> | the object under test calls a series of methods on your mock - not a
> | really
> | informative test and bring of the correct interface issue you
> | mentioned.
> |
> |
> | > Coupled with adequate integration tests this danger
> | > is minimized. Generally I think the tradeoff in increased
> | > testability
> | > makes up for these drawbacks.
> | >
> |
> | +1 for good integration tests; they tend to mitigate that issue.
> |
> |
> | >
> | > Drew
> | >
> | > On Thu, Oct 27, 2011 at 12:49 PM, Jesse Yates
> | > <je...@gmail.com>
> | > wrote:
> | > > Hi all,
> | > >
> | > > Currently Accumulo doesn't have any mocking framework. However, in
> | > > some
> | > > recent work (ACCUMULO-53) doing good testing was greatly
> | > > facilitated by
> | > > adding in a mocking framework. They make it easy to only test the
> | > > piece
> | > of
> | > > the architecture that we are interested in testing, helping to
> | > > isolate
> | > and
> | > > ensure that we are really fixing problems in the code.
> | > >
> | > > So, I would like to propose adding the following test-time
> | > > dependencies
> | > for
> | > > testing:
> | > >
> | > > EasyMock (http://easymock.org/)
> | > > This is a really popular mocking framework for java and is really
> | > > easy to
> | > > use. I've rarely run into a situation that wasn't covered with
> | > > what
> | > EasyMock
> | > > does.
> | > >
> | > > PowerMock (http://code.google.com/p/powermock/)
> | > > This is the utility to cover what other mocking frameworks
> | > > (including
> | > > Mockito) can't do - capturing creation of objects (new
> | > > Whatever(...)) and
> | > > returning mocks instead, mocking static calls, mocking standard
> | > > java lib
> | > > calls, and accessing private methods/fields.
> | > >
> | > > This combination has proven to be incredibly effective and easy to
> | > > use in
> | > > the past, allowing for a really comprehensive testing suite.
> | > >
> | > >  If you want to see an example of this stuff in action, check out
> | > >  the
> | > patch
> | > > 'java_ACCUMULO-53_v2.patch' on ACCUMULO-53.
> | > >
> | > > What does everyone think?
> | > >
> | > > Thanks!
> | > > --Jesse
> | > >
> | > > -------------------
> | > > Jesse Yates
> | > > 240-888-2200
> | > > @jesse_yates
> | > >
> | >
> |
> |
> |
> | --
> | -------------------
> | Jesse Yates
> | 240-888-2200
> | @jesse_yates
>



-- 
-------------------
Jesse Yates
240-888-2200
@jesse_yates

Re: Mocking framework

[Billie J Rinaldi <bi...@ugov.gov>]

On Tuesday, November 1, 2011 3:38:22 PM, "Jesse Yates" <je...@gmail.com> wrote:
> Agreed that for 1 test, its not really worth it. However, the point
> would
> be to establish these frameworks as really part of how testing is done
> going forward.
> 
> So, yeah, 1.5 seems like the better place to put that part of the
> patch.
> I'll break up ACCUMULO-19 into just the fix for BatchDeleter and then
> add
> tickets for adding mocking and another for testing for the batch
> deleter.
> 
> Sound good?

Sounds good!

Billie

Re: Mocking framework

[John W Vines <jo...@ugov.gov>]

+1

----- Original Message -----
| From: "Jesse Yates" <je...@gmail.com>
| To: accumulo-dev@incubator.apache.org
| Sent: Tuesday, November 1, 2011 3:38:22 PM
| Subject: Re: Mocking framework
| Agreed that for 1 test, its not really worth it. However, the point
| would
| be to establish these frameworks as really part of how testing is done
| going forward.
| 
| So, yeah, 1.5 seems like the better place to put that part of the
| patch.
| I'll break up ACCUMULO-19 into just the fix for BatchDeleter and then
| add
| tickets for adding mocking and another for testing for the batch
| deleter.
| 
| Sound good?
| 
| -- Jesse
| -------------------
| Jesse Yates
| 240-888-2200
| @jesse_yates
| 
| On Tue, Nov 1, 2011 at 12:24 PM, Billie J Rinaldi
| <billie.j.rinaldi@ugov.gov
| > wrote:
| 
| > On Monday, October 31, 2011 11:52:11 PM, "Jesse Yates" <
| > jesse.k.yates@gmail.com> wrote:
| > > Are there any objections to adding EasyMock and PowerMock as the
| > > mocking frameworks?
| >
| > I am reluctant to add new dependencies for a single test, which
| > would be
| > the case in 1.4 at this point. If you (or anyone) would be
| > interested in
| > making substantial improvements to 1.5 by adding new tests and/or
| > improving
| > existing tests using these mocking frameworks, I think that would be
| > very
| > useful and would provide sufficient justification for the new
| > dependencies.
| >  (A 1.4 branch will be created soon, perhaps tomorrow, and the trunk
| >  will
| > become 1.5.)
| >
| > Billie
| >

Re: Mocking framework

[Jesse Yates <je...@gmail.com>]

Agreed that for 1 test, its not really worth it. However, the point would
be to establish these frameworks as really part of how testing is done
going forward.

So, yeah, 1.5 seems like the better place to put that part of the patch.
I'll break up ACCUMULO-19 into just the fix for BatchDeleter and then add
tickets for adding mocking and another for testing for the batch deleter.

Sound good?

-- Jesse
-------------------
Jesse Yates
240-888-2200
@jesse_yates

On Tue, Nov 1, 2011 at 12:24 PM, Billie J Rinaldi <billie.j.rinaldi@ugov.gov
> wrote:

> On Monday, October 31, 2011 11:52:11 PM, "Jesse Yates" <
> jesse.k.yates@gmail.com> wrote:
> > Are there any objections to adding EasyMock and PowerMock as the
> > mocking frameworks?
>
> I am reluctant to add new dependencies for a single test, which would be
> the case in 1.4 at this point.  If you (or anyone) would be interested in
> making substantial improvements to 1.5 by adding new tests and/or improving
> existing tests using these mocking frameworks, I think that would be very
> useful and would provide sufficient justification for the new dependencies.
>  (A 1.4 branch will be created soon, perhaps tomorrow, and the trunk will
> become 1.5.)
>
> Billie
>

Re: Mocking framework

[Billie J Rinaldi <bi...@ugov.gov>]

On Monday, October 31, 2011 11:52:11 PM, "Jesse Yates" <je...@gmail.com> wrote:
> Are there any objections to adding EasyMock and PowerMock as the
> mocking frameworks?

I am reluctant to add new dependencies for a single test, which would be the case in 1.4 at this point.  If you (or anyone) would be interested in making substantial improvements to 1.5 by adding new tests and/or improving existing tests using these mocking frameworks, I think that would be very useful and would provide sufficient justification for the new dependencies.  (A 1.4 branch will be created soon, perhaps tomorrow, and the trunk will become 1.5.)

Billie

Re: Mocking framework

[Jesse Yates <je...@gmail.com>]

Are there any objections to adding EasyMock and PowerMock as the mocking
frameworks?

If not, in the next couple days I would like to integrate that change with
ACCUMULO-53 (or I can open up a new ticket if people prefer).

Thanks!

--Jesse
-------------------
Jesse Yates
240-888-2200
@jesse_yates

On Fri, Oct 28, 2011 at 12:24 PM, Jesse Yates <je...@gmail.com>wrote:

> Sorry for the divergent messages - couldnt find a good way to combine them.
>
>
> On Fri, Oct 28, 2011 at 11:54 AM, Keith Turner <ke...@deenlo.com> wrote:
>
>> On Fri, Oct 28, 2011 at 11:19 AM, John W Vines <jo...@ugov.gov>
>> wrote:
>> >
>>
> > 3. What I want to see us doing, at a very high level, is to have the
>> ability to mock an entire TServer to the extent where we will use something
>> to replace Zookeeper (We should probably turn our ZK work with an
>> interface) with a MockZookeeper (not generated through a Mock util) which
>> is nothing more than a Map. Same thing with the FileReader, except a
>> SortedMap, the loggers, and the master. This way we could fully implement a
>> whole TServer without worry about HDFS and Zookeeper.
>>
> To a similar extent I would like to see this done for all core components,
>> but mocking the various connectors we use to get done what we need to. I
>> see a few sets of Mock class we will have to create. But with less chance
>> of divergence in behavior then we currently experience with our
>> MockAccumulo setup.
>> >
>> This sounds like a good goal.  Seems like in addition to mocking, we
>> would need to refactor code to support dependency injection.
>>
>
> With PowerMock we can avoid having to refactor everything since we can
> catch object creation and return your own mock object. Dependency injection
> then becomes a moot need unless we start using a configuration framework
> for launching different elements.
>
>
> --
> -------------------
> Jesse Yates
> 240-888-2200
> @jesse_yates
>
>

Re: Mocking framework

[Jesse Yates <je...@gmail.com>]

Sorry for the divergent messages - couldnt find a good way to combine them.

On Fri, Oct 28, 2011 at 11:54 AM, Keith Turner <ke...@deenlo.com> wrote:

> On Fri, Oct 28, 2011 at 11:19 AM, John W Vines <jo...@ugov.gov>
> wrote:
> >
>
> 3. What I want to see us doing, at a very high level, is to have the
> ability to mock an entire TServer to the extent where we will use something
> to replace Zookeeper (We should probably turn our ZK work with an interface)
> with a MockZookeeper (not generated through a Mock util) which is nothing
> more than a Map. Same thing with the FileReader, except a SortedMap, the
> loggers, and the master. This way we could fully implement a whole TServer
> without worry about HDFS and Zookeeper.
>
To a similar extent I would like to see this done for all core components,
> but mocking the various connectors we use to get done what we need to. I see
> a few sets of Mock class we will have to create. But with less chance of
> divergence in behavior then we currently experience with our MockAccumulo
> setup.
> >
> This sounds like a good goal.  Seems like in addition to mocking, we
> would need to refactor code to support dependency injection.
>

With PowerMock we can avoid having to refactor everything since we can catch
object creation and return your own mock object. Dependency injection then
becomes a moot need unless we start using a configuration framework for
launching different elements.

-- 
-------------------
Jesse Yates
240-888-2200
@jesse_yates

Re: Mocking framework

[Keith Turner <ke...@deenlo.com>]

On Fri, Oct 28, 2011 at 11:19 AM, John W Vines <jo...@ugov.gov> wrote:
>
> 3. What I want to see us doing, at a very high level, is to have the ability to mock an entire TServer to the extent where we will use something to replace Zookeeper (We should probably turn our ZK work with an interface) with a MockZookeeper (not generated through a Mock util) which is nothing more than a Map. Same thing with the FileReader, except a SortedMap, the loggers, and the master. This way we could fully implement a whole TServer without worry about HDFS and Zookeeper. To a similar extent I would like to see this done for all core components, but mocking the various connectors we use to get done what we need to. I see a few sets of Mock class we will have to create. But with less chance of divergence in behavior then we currently experience with our MockAccumulo setup.
>
This sounds like a good goal.  Seems like in addition to mocking, we
would need to refactor code to support dependency injection.

Re: Mocking framework

[Drew Farris <dr...@apache.org>]

This all looks reasonable to me. I wonder if most mocking tools don't
support programmatic definition of return values because it makes the
declaration of expectations less explicit.

Drew

On Fri, Oct 28, 2011 at 11:19 AM, John W Vines <jo...@ugov.gov> wrote:
> I will start off by saying that myself, as well as most of the committers, are no familiar with EasyMock, PowerMock, or really any Mocking tools. But I've done some reading to determine what the scope of mocking can get us. So pardon me if I bore you because I'm just going to lay out what I know in hopes of A. making sure my understanding is correct and B. to see if what I think we should use it for is within the scope of Mocking as well as feasible.
>
> 1. With exception for PowerMock, mocking tools are nothing more than tools which give the ability to define a pre-set set of responses to an interface. That is, if we wanted to implement a low level iterator to replace the Reader we use to read items off disk/out of the in memory map, we could, but we could only give it a set of responses to give when queried. We could not (through just the mocking tools) set it up to give programmatic responses, such as returning the result of another method, unless we 'pre-load' the method result. That is, we couldn't have one mock method which put an item in a map and another mock method pull it out again, unless we define the pull method to return that specific value for the Xth call to the pull method to which that correlated.
>
> 2. PowerMock lets us essentially do the same as things like EasyMock, except we're not bound to interfaces. This way we can use a defined class and only overwrite the methods we want with predefined results.
>
> 3. What I want to see us doing, at a very high level, is to have the ability to mock an entire TServer to the extent where we will use something to replace Zookeeper (We should probably turn our ZK work with an interface) with a MockZookeeper (not generated through a Mock util) which is nothing more than a Map. Same thing with the FileReader, except a SortedMap, the loggers, and the master. This way we could fully implement a whole TServer without worry about HDFS and Zookeeper. To a similar extent I would like to see this done for all core components, but mocking the various connectors we use to get done what we need to. I see a few sets of Mock class we will have to create. But with less chance of divergence in behavior then we currently experience with our MockAccumulo setup.
>
> 4. My principal concern about 3 above is divergence with the code (less so than our current setup but with emulating thrift interfaces we could get problems). But also the feasibility of mocking thrift connections. But if all this works out, I think we'll be able to optimize our code coverage better than our current setup without the requirement to do a full Accumulo setup for junit tests.
>
> Cheers
> John
>
> ----- Original Message -----
> | From: "Jesse Yates" <je...@gmail.com>
> | To: accumulo-dev@incubator.apache.org
> | Sent: Thursday, October 27, 2011 2:21:33 PM
> | Subject: Re: Mocking framework
> | On Thu, Oct 27, 2011 at 10:55 AM, Drew Farris <dr...@apache.org> wrote:
> |
> | > I agree in principle without having reviewed the tests accompanying
> | > ACCUMULO-53.
> | >
> | > I've used EasyMock in the past and have been very happy with the
> | > ability to programatically mock implementations of interfaces,
> | > define
> | > test input and assert a proper sequence of calls, and output. In
> | > general I think mock object testing is a useful tool, especially in
> | > cases where the existing interface implementations have heavyweight
> | > external dependencies, such as for mocking JDBC result sets or
> | > portions of the Hadoop framework. I've used mock objects to write
> | > unit
> | > tests for mappers and reducers by mocking out the hadoop context,
> | > counters, etc. Arguably this would be have been better achieved with
> | > MRUnit, which wasn't on my radar at the time.
> | >
> | > I found that I didn't have to jump through hoops to make code
> | > testable. Private methods could really be private instead of
> | > protected
> | > or public in order to facilitate testing.
> |
> |
> | The comment about testing private methods was only a comment on the
> | power of
> | the framework, not on what should be done. IMHO you shouldn't be
> | testing
> | private methods w/o a very good reason.
> |
> |
> | > I also appreciate the
> | > ability to not only assert return values, but assert a series of
> | > method calls and side effects.
> | >
> | > The danger with this is only testing in isolation using mocks -- If
> | > the mocks don't follow the contract or conventions the
> | > implementations
> | > the tests are useless or deceptive. Implementation class behavior
> | > can
> | > change in ways that are invisible from an api perspective, so it's
> | > important that these sorts of changes be reflected adequately by the
> | > mocks themselves.
> |
> |
> | Of course there is also the case where you end up just checking to
> | make sure
> | the object under test calls a series of methods on your mock - not a
> | really
> | informative test and bring of the correct interface issue you
> | mentioned.
> |
> |
> | > Coupled with adequate integration tests this danger
> | > is minimized. Generally I think the tradeoff in increased
> | > testability
> | > makes up for these drawbacks.
> | >
> |
> | +1 for good integration tests; they tend to mitigate that issue.
> |
> |
> | >
> | > Drew
> | >
> | > On Thu, Oct 27, 2011 at 12:49 PM, Jesse Yates
> | > <je...@gmail.com>
> | > wrote:
> | > > Hi all,
> | > >
> | > > Currently Accumulo doesn't have any mocking framework. However, in
> | > > some
> | > > recent work (ACCUMULO-53) doing good testing was greatly
> | > > facilitated by
> | > > adding in a mocking framework. They make it easy to only test the
> | > > piece
> | > of
> | > > the architecture that we are interested in testing, helping to
> | > > isolate
> | > and
> | > > ensure that we are really fixing problems in the code.
> | > >
> | > > So, I would like to propose adding the following test-time
> | > > dependencies
> | > for
> | > > testing:
> | > >
> | > > EasyMock (http://easymock.org/)
> | > > This is a really popular mocking framework for java and is really
> | > > easy to
> | > > use. I've rarely run into a situation that wasn't covered with
> | > > what
> | > EasyMock
> | > > does.
> | > >
> | > > PowerMock (http://code.google.com/p/powermock/)
> | > > This is the utility to cover what other mocking frameworks
> | > > (including
> | > > Mockito) can't do - capturing creation of objects (new
> | > > Whatever(...)) and
> | > > returning mocks instead, mocking static calls, mocking standard
> | > > java lib
> | > > calls, and accessing private methods/fields.
> | > >
> | > > This combination has proven to be incredibly effective and easy to
> | > > use in
> | > > the past, allowing for a really comprehensive testing suite.
> | > >
> | > >  If you want to see an example of this stuff in action, check out
> | > >  the
> | > patch
> | > > 'java_ACCUMULO-53_v2.patch' on ACCUMULO-53.
> | > >
> | > > What does everyone think?
> | > >
> | > > Thanks!
> | > > --Jesse
> | > >
> | > > -------------------
> | > > Jesse Yates
> | > > 240-888-2200
> | > > @jesse_yates
> | > >
> | >
> |
> |
> |
> | --
> | -------------------
> | Jesse Yates
> | 240-888-2200
> | @jesse_yates
>

Re: Mocking framework

[John W Vines <jo...@ugov.gov>]

I will start off by saying that myself, as well as most of the committers, are no familiar with EasyMock, PowerMock, or really any Mocking tools. But I've done some reading to determine what the scope of mocking can get us. So pardon me if I bore you because I'm just going to lay out what I know in hopes of A. making sure my understanding is correct and B. to see if what I think we should use it for is within the scope of Mocking as well as feasible.

1. With exception for PowerMock, mocking tools are nothing more than tools which give the ability to define a pre-set set of responses to an interface. That is, if we wanted to implement a low level iterator to replace the Reader we use to read items off disk/out of the in memory map, we could, but we could only give it a set of responses to give when queried. We could not (through just the mocking tools) set it up to give programmatic responses, such as returning the result of another method, unless we 'pre-load' the method result. That is, we couldn't have one mock method which put an item in a map and another mock method pull it out again, unless we define the pull method to return that specific value for the Xth call to the pull method to which that correlated.

2. PowerMock lets us essentially do the same as things like EasyMock, except we're not bound to interfaces. This way we can use a defined class and only overwrite the methods we want with predefined results.

3. What I want to see us doing, at a very high level, is to have the ability to mock an entire TServer to the extent where we will use something to replace Zookeeper (We should probably turn our ZK work with an interface) with a MockZookeeper (not generated through a Mock util) which is nothing more than a Map. Same thing with the FileReader, except a SortedMap, the loggers, and the master. This way we could fully implement a whole TServer without worry about HDFS and Zookeeper. To a similar extent I would like to see this done for all core components, but mocking the various connectors we use to get done what we need to. I see a few sets of Mock class we will have to create. But with less chance of divergence in behavior then we currently experience with our MockAccumulo setup.

4. My principal concern about 3 above is divergence with the code (less so than our current setup but with emulating thrift interfaces we could get problems). But also the feasibility of mocking thrift connections. But if all this works out, I think we'll be able to optimize our code coverage better than our current setup without the requirement to do a full Accumulo setup for junit tests.

Cheers
John

----- Original Message -----
| From: "Jesse Yates" <je...@gmail.com>
| To: accumulo-dev@incubator.apache.org
| Sent: Thursday, October 27, 2011 2:21:33 PM
| Subject: Re: Mocking framework
| On Thu, Oct 27, 2011 at 10:55 AM, Drew Farris <dr...@apache.org> wrote:
| 
| > I agree in principle without having reviewed the tests accompanying
| > ACCUMULO-53.
| >
| > I've used EasyMock in the past and have been very happy with the
| > ability to programatically mock implementations of interfaces,
| > define
| > test input and assert a proper sequence of calls, and output. In
| > general I think mock object testing is a useful tool, especially in
| > cases where the existing interface implementations have heavyweight
| > external dependencies, such as for mocking JDBC result sets or
| > portions of the Hadoop framework. I've used mock objects to write
| > unit
| > tests for mappers and reducers by mocking out the hadoop context,
| > counters, etc. Arguably this would be have been better achieved with
| > MRUnit, which wasn't on my radar at the time.
| >
| > I found that I didn't have to jump through hoops to make code
| > testable. Private methods could really be private instead of
| > protected
| > or public in order to facilitate testing.
| 
| 
| The comment about testing private methods was only a comment on the
| power of
| the framework, not on what should be done. IMHO you shouldn't be
| testing
| private methods w/o a very good reason.
| 
| 
| > I also appreciate the
| > ability to not only assert return values, but assert a series of
| > method calls and side effects.
| >
| > The danger with this is only testing in isolation using mocks -- If
| > the mocks don't follow the contract or conventions the
| > implementations
| > the tests are useless or deceptive. Implementation class behavior
| > can
| > change in ways that are invisible from an api perspective, so it's
| > important that these sorts of changes be reflected adequately by the
| > mocks themselves.
| 
| 
| Of course there is also the case where you end up just checking to
| make sure
| the object under test calls a series of methods on your mock - not a
| really
| informative test and bring of the correct interface issue you
| mentioned.
| 
| 
| > Coupled with adequate integration tests this danger
| > is minimized. Generally I think the tradeoff in increased
| > testability
| > makes up for these drawbacks.
| >
| 
| +1 for good integration tests; they tend to mitigate that issue.
| 
| 
| >
| > Drew
| >
| > On Thu, Oct 27, 2011 at 12:49 PM, Jesse Yates
| > <je...@gmail.com>
| > wrote:
| > > Hi all,
| > >
| > > Currently Accumulo doesn't have any mocking framework. However, in
| > > some
| > > recent work (ACCUMULO-53) doing good testing was greatly
| > > facilitated by
| > > adding in a mocking framework. They make it easy to only test the
| > > piece
| > of
| > > the architecture that we are interested in testing, helping to
| > > isolate
| > and
| > > ensure that we are really fixing problems in the code.
| > >
| > > So, I would like to propose adding the following test-time
| > > dependencies
| > for
| > > testing:
| > >
| > > EasyMock (http://easymock.org/)
| > > This is a really popular mocking framework for java and is really
| > > easy to
| > > use. I've rarely run into a situation that wasn't covered with
| > > what
| > EasyMock
| > > does.
| > >
| > > PowerMock (http://code.google.com/p/powermock/)
| > > This is the utility to cover what other mocking frameworks
| > > (including
| > > Mockito) can't do - capturing creation of objects (new
| > > Whatever(...)) and
| > > returning mocks instead, mocking static calls, mocking standard
| > > java lib
| > > calls, and accessing private methods/fields.
| > >
| > > This combination has proven to be incredibly effective and easy to
| > > use in
| > > the past, allowing for a really comprehensive testing suite.
| > >
| > >  If you want to see an example of this stuff in action, check out
| > >  the
| > patch
| > > 'java_ACCUMULO-53_v2.patch' on ACCUMULO-53.
| > >
| > > What does everyone think?
| > >
| > > Thanks!
| > > --Jesse
| > >
| > > -------------------
| > > Jesse Yates
| > > 240-888-2200
| > > @jesse_yates
| > >
| >
| 
| 
| 
| --
| -------------------
| Jesse Yates
| 240-888-2200
| @jesse_yates

Re: Mocking framework

[Jesse Yates <je...@gmail.com>]

On Thu, Oct 27, 2011 at 10:55 AM, Drew Farris <dr...@apache.org> wrote:

> I agree in principle without having reviewed the tests accompanying
> ACCUMULO-53.
>
> I've used EasyMock in the past and have been very happy with the
> ability to programatically mock implementations of interfaces, define
> test input and assert a proper sequence of calls, and output. In
> general I think mock object testing is a useful tool, especially in
> cases where the existing interface implementations have heavyweight
> external dependencies, such as for mocking JDBC result sets or
> portions of the Hadoop framework. I've used mock objects to write unit
> tests for mappers and reducers by mocking out the hadoop context,
> counters, etc. Arguably this would be have been better achieved with
> MRUnit, which wasn't on my radar at the time.
>
> I found that I didn't have to jump through hoops to make code
> testable. Private methods could really be private instead of protected
> or public in order to facilitate testing.


The comment about testing private methods was only a comment on the power of
the framework, not on what should be done. IMHO you shouldn't be testing
private methods w/o a very good reason.


> I also appreciate the
> ability to not only assert return values, but assert a series of
> method calls and side effects.
>
> The danger with this is only testing in isolation using mocks -- If
> the mocks don't follow the contract or conventions the implementations
> the tests are useless or deceptive. Implementation class behavior can
> change in ways that are invisible from an api perspective, so it's
> important that these sorts of changes be reflected adequately by the
> mocks themselves.


Of course there is also the case where you end up just checking to make sure
the object under test calls a series of methods on your mock - not a really
informative test and bring of the correct interface issue you mentioned.


> Coupled with adequate integration tests this danger
> is minimized. Generally I think the tradeoff in increased testability
> makes up for these drawbacks.
>

+1 for good integration tests; they tend to mitigate that issue.


>
> Drew
>
> On Thu, Oct 27, 2011 at 12:49 PM, Jesse Yates <je...@gmail.com>
> wrote:
> > Hi all,
> >
> > Currently Accumulo doesn't have any mocking framework. However, in some
> > recent work (ACCUMULO-53) doing good testing was greatly facilitated by
> > adding in a mocking framework. They make it easy to only test the piece
> of
> > the architecture that we are interested in testing, helping to isolate
> and
> > ensure that we are really fixing problems in the code.
> >
> > So, I would like to propose adding the following test-time dependencies
> for
> > testing:
> >
> > EasyMock (http://easymock.org/)
> > This is a really popular mocking framework for java and is really easy to
> > use. I've rarely run into a situation that wasn't covered with what
> EasyMock
> > does.
> >
> > PowerMock (http://code.google.com/p/powermock/)
> > This is the utility to cover what other mocking frameworks (including
> > Mockito) can't do - capturing creation of objects (new Whatever(...)) and
> > returning mocks instead, mocking static calls, mocking standard java lib
> > calls, and accessing private methods/fields.
> >
> > This combination has proven to be incredibly effective and easy to use in
> > the past, allowing for a really comprehensive testing suite.
> >
> >  If you want to see an example of this stuff in action, check out the
> patch
> > 'java_ACCUMULO-53_v2.patch' on ACCUMULO-53.
> >
> > What does everyone think?
> >
> > Thanks!
> > --Jesse
> >
> > -------------------
> > Jesse Yates
> > 240-888-2200
> > @jesse_yates
> >
>



-- 
-------------------
Jesse Yates
240-888-2200
@jesse_yates

Re: Mocking framework

[Drew Farris <dr...@apache.org>]

I agree in principle without having reviewed the tests accompanying ACCUMULO-53.

I've used EasyMock in the past and have been very happy with the
ability to programatically mock implementations of interfaces, define
test input and assert a proper sequence of calls, and output. In
general I think mock object testing is a useful tool, especially in
cases where the existing interface implementations have heavyweight
external dependencies, such as for mocking JDBC result sets or
portions of the Hadoop framework. I've used mock objects to write unit
tests for mappers and reducers by mocking out the hadoop context,
counters, etc. Arguably this would be have been better achieved with
MRUnit, which wasn't on my radar at the time.

I found that I didn't have to jump through hoops to make code
testable. Private methods could really be private instead of protected
or public in order to facilitate testing. I also appreciate the
ability to not only assert return values, but assert a series of
method calls and side effects.

The danger with this is only testing in isolation using mocks -- If
the mocks don't follow the contract or conventions the implementations
the tests are useless or deceptive. Implementation class behavior can
change in ways that are invisible from an api perspective, so it's
important that these sorts of changes be reflected adequately by the
mocks themselves. Coupled with adequate integration tests this danger
is minimized. Generally I think the tradeoff in increased testability
makes up for these drawbacks.

Drew

On Thu, Oct 27, 2011 at 12:49 PM, Jesse Yates <je...@gmail.com> wrote:
> Hi all,
>
> Currently Accumulo doesn't have any mocking framework. However, in some
> recent work (ACCUMULO-53) doing good testing was greatly facilitated by
> adding in a mocking framework. They make it easy to only test the piece of
> the architecture that we are interested in testing, helping to isolate and
> ensure that we are really fixing problems in the code.
>
> So, I would like to propose adding the following test-time dependencies for
> testing:
>
> EasyMock (http://easymock.org/)
> This is a really popular mocking framework for java and is really easy to
> use. I've rarely run into a situation that wasn't covered with what EasyMock
> does.
>
> PowerMock (http://code.google.com/p/powermock/)
> This is the utility to cover what other mocking frameworks (including
> Mockito) can't do - capturing creation of objects (new Whatever(...)) and
> returning mocks instead, mocking static calls, mocking standard java lib
> calls, and accessing private methods/fields.
>
> This combination has proven to be incredibly effective and easy to use in
> the past, allowing for a really comprehensive testing suite.
>
>  If you want to see an example of this stuff in action, check out the patch
> 'java_ACCUMULO-53_v2.patch' on ACCUMULO-53.
>
> What does everyone think?
>
> Thanks!
> --Jesse
>
> -------------------
> Jesse Yates
> 240-888-2200
> @jesse_yates
>