Where are we with Surefire 2.4?

[Dan Fabulich <da...@fabulich.com>]

As you should all know by now, I've been on a hard sprint most of this 
week to get a reasonable-looking Surefire 2.4 out the door.  I think we're 
basically where I want it to be.

Just three main areas remain.

1) useSystemClassLoader issues
2) Snapshot dependencies
3) Measuring code coverage
and finally a remark 4) Dan takes a breather

1) useSystemClassLoader issues

Notably a lot of the very trickiest Surefire bugs we've gotten have all 
been "Not Reproducible" or "Won't Fix", because all you have to do is set 
useSystemClassLoader=true and things "just work."  I'd love to make that 
the default setting for Surefire, but there's some stuff in the way that 
worries me about doing that.

SUREFIRE-334 (targeted for 2.3.1) points out that we're depending on a 
good chunk of plexus stuff in Surefire, and that this would infect the AUT 
if we just set useSystemClassLoader=true.  332 and 347 are probably dupes 
of this issue, but these things are subtle so I haven't closed them yet.

Jason suggested that we might use the maven-shade-plugin to shade our 
plexus dependencies (repackaging them in different package names and 
updating dependencies); it's a good idea, but maven-shade-plugin fails to 
work properly on the plexus archiver (it doesn't translate interfaces 
correctly, apparently?).

It would be great to resolve this issue and make useSystemClassLoader the 
default BEFORE 2.4 ships (it *is* targeted for 2.3.1).  I think Brett has 
claimed these.  (The other 2.3.1 bug 335 is a weird one, suggesting that 
we might somehow useSystemClassLoader without forking; that sounds like 
nonsense to me and I'd be curious to hear someone [Brett] comment on it.)

2) Snapshot dependencies (SUREFIRE-394)

2.4 depends on plexus-utils 1.4.8-SNAPSHOT; it should be released before 
we release Surefire 2.4.

Up until last week or so we used to depend on plexus-archiver 
alpha-10-SNAPSHOT, but that created ICCEs (see SUREFIRE-389) and so I 
downgraded to alpha-7. If we DO need alpha-10-SNAPSHOT, then we'll need to 
figure out the dependency issue (perhaps by fixing SUREFIRE-334 and 
applying maven-shade-plugin?).  jdcasey apparently knew why we needed 
alpha-10, but I still don't; my integration tests are all passing without 
it.

3) Measuring code coverage

I wrote all these tests!  How good are they?  Measuring the coverage of 
out-of-proc integration tests seems to be pretty tricky, though not 
impossible.  Any ideas?

4) Dan takes a breather

I've been focused on getting this far all week, but I'll be basically 
useless for most of the next week, maybe two.  I'll try to be available to 
answer e-mail and chat, but this next week I won't be doing a lot of 
coding.

I'd appreciate it if somebody non-me were to take up the baton and help 
get this release out sometime this week (we're so close)! :-)

OK, that's it.  Let me know if you have questions.  I'm off to bed!

-Dan

Re: Where are we with Surefire 2.4?

[Dan Fabulich <da...@fabulich.com>]

Brett Porter wrote:

> As for p-archiver, I'm inclined to try and remove it since it's just 
> assembling a very basic JAR.

Beware... manifest classpaths are surprisingly tricky to construct.  They 
have a lot of weird corner cases you have to take care of.  (They must be 
wrapped, wrapped lines must start with a space, but they're 
space-delimited...)

But, yeah, I can't really object to removing it if that's all we need it 
for.  We ARE using plexus-utils in a much more sophisticated way, (though 
I bet it would be more amenable to the shade plugin...?) and might not 
want to drop it.

Plus I just asked olamy to release a new version of plexus-utils for 
Surefire, so I'd feel kinda bad if we turned around and dropped it.  ;-)

-Dan

Re: Where are we with Surefire 2.4?

[Brett Porter <br...@apache.org>]

Hey Dan,

I concur on (1) and should have cycles this weekend on the flight  
back to wrap up 2.3.1 and hopefully take care of this for 2.4. You're  
right, the use of the two options together is non-sensical - which I  
think the issue was to just make it not attempt to let you use them  
together? As for p-archiver, I'm inclined to try and remove it since  
it's just assembling a very basic JAR.

For (2), as above.

I'll reply separately on (3) when I get a chance.

+1 for (4) - your efforts are very very much appreciated. Thanks!

Cheers,
Brett

On 26/11/2007, at 7:23 PM, Dan Fabulich wrote:

>
> As you should all know by now, I've been on a hard sprint most of  
> this week to get a reasonable-looking Surefire 2.4 out the door.  I  
> think we're basically where I want it to be.
>
> Just three main areas remain.
>
> 1) useSystemClassLoader issues
> 2) Snapshot dependencies
> 3) Measuring code coverage
> and finally a remark 4) Dan takes a breather
>
> 1) useSystemClassLoader issues
>
> Notably a lot of the very trickiest Surefire bugs we've gotten have  
> all been "Not Reproducible" or "Won't Fix", because all you have to  
> do is set useSystemClassLoader=true and things "just work."  I'd  
> love to make that the default setting for Surefire, but there's  
> some stuff in the way that worries me about doing that.
>
> SUREFIRE-334 (targeted for 2.3.1) points out that we're depending  
> on a good chunk of plexus stuff in Surefire, and that this would  
> infect the AUT if we just set useSystemClassLoader=true.  332 and  
> 347 are probably dupes of this issue, but these things are subtle  
> so I haven't closed them yet.
>
> Jason suggested that we might use the maven-shade-plugin to shade  
> our plexus dependencies (repackaging them in different package  
> names and updating dependencies); it's a good idea, but maven-shade- 
> plugin fails to work properly on the plexus archiver (it doesn't  
> translate interfaces correctly, apparently?).
>
> It would be great to resolve this issue and make  
> useSystemClassLoader the default BEFORE 2.4 ships (it *is* targeted  
> for 2.3.1).  I think Brett has claimed these.  (The other 2.3.1 bug  
> 335 is a weird one, suggesting that we might somehow  
> useSystemClassLoader without forking; that sounds like nonsense to  
> me and I'd be curious to hear someone [Brett] comment on it.)
>
> 2) Snapshot dependencies (SUREFIRE-394)
>
> 2.4 depends on plexus-utils 1.4.8-SNAPSHOT; it should be released  
> before we release Surefire 2.4.
>
> Up until last week or so we used to depend on plexus-archiver  
> alpha-10-SNAPSHOT, but that created ICCEs (see SUREFIRE-389) and so  
> I downgraded to alpha-7. If we DO need alpha-10-SNAPSHOT, then  
> we'll need to figure out the dependency issue (perhaps by fixing  
> SUREFIRE-334 and applying maven-shade-plugin?).  jdcasey apparently  
> knew why we needed alpha-10, but I still don't; my integration  
> tests are all passing without it.
>
> 3) Measuring code coverage
>
> I wrote all these tests!  How good are they?  Measuring the  
> coverage of out-of-proc integration tests seems to be pretty  
> tricky, though not impossible.  Any ideas?
>
> 4) Dan takes a breather
>
> I've been focused on getting this far all week, but I'll be  
> basically useless for most of the next week, maybe two.  I'll try  
> to be available to answer e-mail and chat, but this next week I  
> won't be doing a lot of coding.
>
> I'd appreciate it if somebody non-me were to take up the baton and  
> help get this release out sometime this week (we're so close)! :-)
>
> OK, that's it.  Let me know if you have questions.  I'm off to bed!
>
> -Dan

--
Brett Porter - brett@apache.org
Blog: http://www.devzuz.org/blogs/bporter/

Re: Where are we with Surefire 2.4?

[Dan Fabulich <da...@fabulich.com>]

I forgot, there's one more issue that should really get into Surefire 2.4: 
SUREFIRE-303 http://jira.codehaus.org/browse/SUREFIRE-303 "Ignored/Skipped 
tests are not reported"

Currently there's no standard on the ground for how JUnit 4 @Ignores 
should be handled.  TestNG has a reasonable example (they put <skipped> 
elements into the JUnit XML); we should, I suppose, follow their lead, 
though I'm afraid the Ant/JUnit teams will decide to call it <ignored> 
instead...

Still, I think we'd better clarify this before we try to support something 
non-standard in Surefire 2.4.

-Dan

Dan Fabulich wrote:

>
> As you should all know by now, I've been on a hard sprint most of this week 
> to get a reasonable-looking Surefire 2.4 out the door.  I think we're 
> basically where I want it to be.
>
> Just three main areas remain.
>
> 1) useSystemClassLoader issues
> 2) Snapshot dependencies
> 3) Measuring code coverage
> and finally a remark 4) Dan takes a breather
>
> 1) useSystemClassLoader issues
>
> Notably a lot of the very trickiest Surefire bugs we've gotten have all been 
> "Not Reproducible" or "Won't Fix", because all you have to do is set 
> useSystemClassLoader=true and things "just work."  I'd love to make that the 
> default setting for Surefire, but there's some stuff in the way that worries 
> me about doing that.
>
> SUREFIRE-334 (targeted for 2.3.1) points out that we're depending on a good 
> chunk of plexus stuff in Surefire, and that this would infect the AUT if we 
> just set useSystemClassLoader=true.  332 and 347 are probably dupes of this 
> issue, but these things are subtle so I haven't closed them yet.
>
> Jason suggested that we might use the maven-shade-plugin to shade our plexus 
> dependencies (repackaging them in different package names and updating 
> dependencies); it's a good idea, but maven-shade-plugin fails to work 
> properly on the plexus archiver (it doesn't translate interfaces correctly, 
> apparently?).
>
> It would be great to resolve this issue and make useSystemClassLoader the 
> default BEFORE 2.4 ships (it *is* targeted for 2.3.1).  I think Brett has 
> claimed these.  (The other 2.3.1 bug 335 is a weird one, suggesting that we 
> might somehow useSystemClassLoader without forking; that sounds like nonsense 
> to me and I'd be curious to hear someone [Brett] comment on it.)
>
> 2) Snapshot dependencies (SUREFIRE-394)
>
> 2.4 depends on plexus-utils 1.4.8-SNAPSHOT; it should be released before we 
> release Surefire 2.4.
>
> Up until last week or so we used to depend on plexus-archiver 
> alpha-10-SNAPSHOT, but that created ICCEs (see SUREFIRE-389) and so I 
> downgraded to alpha-7. If we DO need alpha-10-SNAPSHOT, then we'll need to 
> figure out the dependency issue (perhaps by fixing SUREFIRE-334 and applying 
> maven-shade-plugin?).  jdcasey apparently knew why we needed alpha-10, but I 
> still don't; my integration tests are all passing without it.
>
> 3) Measuring code coverage
>
> I wrote all these tests!  How good are they?  Measuring the coverage of 
> out-of-proc integration tests seems to be pretty tricky, though not 
> impossible.  Any ideas?
>
> 4) Dan takes a breather
>
> I've been focused on getting this far all week, but I'll be basically useless 
> for most of the next week, maybe two.  I'll try to be available to answer 
> e-mail and chat, but this next week I won't be doing a lot of coding.
>
> I'd appreciate it if somebody non-me were to take up the baton and help get 
> this release out sometime this week (we're so close)! :-)
>
> OK, that's it.  Let me know if you have questions.  I'm off to bed!
>
> -Dan
>