Invoking integration tests in test-patch

[Josh Elser <jo...@gmail.com>]

Hi all,

I've had this come up a few times in talks about enabling PreCommit for 
some other projects. Best as I can tell, the support to invoke 
integration tests isn't supported.

Assuming I'm not just wrong, any direction on where to start looking for 
this? I have a general idea of how I think the functionality would fit 
in, but I'm not sure where to start looking in the code. Maybe the 
maven_modules_worker function in maven.sh? (Also, have I missed any docs 
that might have already covered this?)

Thanks in advance!

- Josh

Re: Invoking integration tests in test-patch

[Josh Elser <jo...@gmail.com>]

Allen Wittenauer wrote:
>> On Jan 10, 2016, at 8:18 PM, Josh Elser<jo...@gmail.com>  wrote:
>> Assuming I'm not just wrong, any direction on where to start looking for this? I have a general idea of how I think the functionality would fit in, but I'm not sure where to start looking in the code. Maybe the maven_modules_worker function in maven.sh? (Also, have I missed any docs that might have already covered this?)
>
>
> 	if integration tests are expected to provide some sort of test output to be analyzed similar to the unit tests, then I’d probably start with the unit test code and go from there.  While this is a more difficult path, the benefit is that you’ll likely end up with automatic test format support.

I'll have to double checking the naming convention, but the contents of 
the files are very similar AFAIK.

>
>> On Jan 11, 2016, at 9:20 AM, Sean Busbey<bu...@apache.org>  wrote:
>>
>>
>> https://github.com/apache/yetus/blob/master/precommit/test-patch.d/checkstyle.sh#L164
>>
>> https://github.com/apache/yetus/blob/master/precommit/test-patch.d/findbugs.sh#L100
>>
>> They take different approaches, with findbugs using
>> maven_modules_worker. It looks much simpler to base your test on the
>> findbugs plugin, IMHO.
>
> 	Using modules_workers is *definitely* the way to go.  I think at this point all of the build tool specific plug-ins except for checkstyle use it. There was a reason why I couldn’t do that with checkstyle, but I can’t remember what it was now.  It might have been because ant required the console output setting, but we didn’t have a way to do that at the time.  (We do now and this could should probably get cleaned up to use the standard bits.)
>

Great, thanks for your feedback as well, Allen!

Re: Invoking integration tests in test-patch

[Allen Wittenauer <aw...@altiscale.com>]

> On Jan 10, 2016, at 8:18 PM, Josh Elser <jo...@gmail.com> wrote:
> Assuming I'm not just wrong, any direction on where to start looking for this? I have a general idea of how I think the functionality would fit in, but I'm not sure where to start looking in the code. Maybe the maven_modules_worker function in maven.sh? (Also, have I missed any docs that might have already covered this?)


	if integration tests are expected to provide some sort of test output to be analyzed similar to the unit tests, then I’d probably start with the unit test code and go from there.  While this is a more difficult path, the benefit is that you’ll likely end up with automatic test format support.


> On Jan 11, 2016, at 9:20 AM, Sean Busbey <bu...@apache.org> wrote:
> 
> 
> https://github.com/apache/yetus/blob/master/precommit/test-patch.d/checkstyle.sh#L164
> 
> https://github.com/apache/yetus/blob/master/precommit/test-patch.d/findbugs.sh#L100
> 
> They take different approaches, with findbugs using
> maven_modules_worker. It looks much simpler to base your test on the
> findbugs plugin, IMHO.

	Using modules_workers is *definitely* the way to go.  I think at this point all of the build tool specific plug-ins except for checkstyle use it. There was a reason why I couldn’t do that with checkstyle, but I can’t remember what it was now.  It might have been because ant required the console output setting, but we didn’t have a way to do that at the time.  (We do now and this could should probably get cleaned up to use the standard bits.)

Re: Invoking integration tests in test-patch

[Josh Elser <jo...@gmail.com>]

Sean Busbey wrote:
> I can think of some hacky ways to use the profile yetus tries to
> activate[1] to have the integration tests run during the test-phase.
> But that's a bad idea, since you'll essentially have to duplicate any
> work configured to happen in the pre-integration-test and
> post-integration-test phases.
>
> Probably easiest will be to define a new test plugin that calls the
> verify goal (with the needed params to avoid e.g. unit tests, javadoc,
> etc). The checkstyle and findbugs test plugins both include having
> maven run with goals:
>
> https://github.com/apache/yetus/blob/master/precommit/test-patch.d/checkstyle.sh#L164
>
> https://github.com/apache/yetus/blob/master/precommit/test-patch.d/findbugs.sh#L100

This is exactly what I had in mind. Thanks for the pointer.

> They take different approaches, with findbugs using
> maven_modules_worker. It looks much simpler to base your test on the
> findbugs plugin, IMHO.
>
> We'd love a "make a Yetus test plugin" tutorial based on "run a maven
> plugin," if you wouldn't mind writing down the process you end up
> going through. ;)

I will keep this in mind :D

>
> [1]: http://yetus.apache.org/documentation/0.1.0/precommit-buildtools/#test-profile
>

Re: Invoking integration tests in test-patch

[Sean Busbey <bu...@apache.org>]

I can think of some hacky ways to use the profile yetus tries to
activate[1] to have the integration tests run during the test-phase.
But that's a bad idea, since you'll essentially have to duplicate any
work configured to happen in the pre-integration-test and
post-integration-test phases.

Probably easiest will be to define a new test plugin that calls the
verify goal (with the needed params to avoid e.g. unit tests, javadoc,
etc). The checkstyle and findbugs test plugins both include having
maven run with goals:

https://github.com/apache/yetus/blob/master/precommit/test-patch.d/checkstyle.sh#L164

https://github.com/apache/yetus/blob/master/precommit/test-patch.d/findbugs.sh#L100

They take different approaches, with findbugs using
maven_modules_worker. It looks much simpler to base your test on the
findbugs plugin, IMHO.

We'd love a "make a Yetus test plugin" tutorial based on "run a maven
plugin," if you wouldn't mind writing down the process you end up
going through. ;)


[1]: http://yetus.apache.org/documentation/0.1.0/precommit-buildtools/#test-profile

On Sun, Jan 10, 2016 at 10:18 PM, Josh Elser <jo...@gmail.com> wrote:
> Hi all,
>
> I've had this come up a few times in talks about enabling PreCommit for some
> other projects. Best as I can tell, the support to invoke integration tests
> isn't supported.
>
> Assuming I'm not just wrong, any direction on where to start looking for
> this? I have a general idea of how I think the functionality would fit in,
> but I'm not sure where to start looking in the code. Maybe the
> maven_modules_worker function in maven.sh? (Also, have I missed any docs
> that might have already covered this?)
>
> Thanks in advance!
>
> - Josh