You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Gary Gregory <GG...@seagullsoftware.com> on 2010/04/05 18:16:55 UTC
RE: [codec][lang] Provide a test jar plus [daemon]
Seeing the discussion about [daemon] and not releasing made me think of another use for a test jar file.
What I would like to know when evaluating an RC for releasing a maintenance of a commons component (from x.y.n to x.y.n+1) is that there is 100% binary compatibility.
As part of the build I would run (at least) the 1.0.2 unit tests against the 1.0.3 RC. If 100% pass all is well, if not, it is either a bug or a known acceptable failure fixing a bug and should be documented somehow, probably in a ticket.
This would mean that a release 1.0.3 RC would include foo-test-1.0.2.jar. And maybe also foo-test-1.0.0.jar and foo-test-1.0.1.jar, hm...
Thoughts?
Gary Gregory
Senior Software Engineer
Seagull Software
email: ggregory@seagullsoftware.com
email: ggregory@apache.org
www.seagullsoftware.com
From: Gary Gregory
Sent: Tuesday, March 30, 2010 16:58
To: Commons Developers List
Subject: [codec][lang] Provide a test jar
I am starting with codec and lang since it what I am most interested in ATM...
I would like to run commons.xxx unit tests as part of my build as a sanity check when I try out a new combo of JVM, OS, jars, etc.
Right now, I would have to compile the unit tests as part of my build which is not great.
So, what about providing a commons-codec-1.5-test.jar file like we provide a sources and javadoc file?
Same for any commons project really...
Thoughts?
Gary Gregory
Senior Software Engineer
Seagull Software
email: ggregory@seagullsoftware.com
email: ggregory@apache.org
www.seagullsoftware.com
RE: [codec][lang] Provide a test jar plus [daemon]
Posted by Gary Gregory <GG...@seagullsoftware.com>.
Hi Jörg...
> -----Original Message-----
> From: Jörg Schaible [mailto:joerg.schaible@gmx.de]
> Sent: Tuesday, April 06, 2010 01:17
> To: dev@commons.apache.org
> Subject: RE: [codec][lang] Provide a test jar plus [daemon]
>
> Hi Gary,
>
> Gary Gregory wrote at Monday, 5. April 2010 18:16:
>
> > Seeing the discussion about [daemon] and not releasing made me think of
> > another use for a test jar file.
> >
> > What I would like to know when evaluating an RC for releasing a
> > maintenance of a commons component (from x.y.n to x.y.n+1) is that there
> > is 100% binary compatibility.
> >
> > As part of the build I would run (at least) the 1.0.2 unit tests against
> > the 1.0.3 RC. If 100% pass all is well, if not, it is either a bug or a
> > known acceptable failure fixing a bug and should be documented somehow,
> > probably in a ticket.
> >
> > This would mean that a release 1.0.3 RC would include foo-test-1.0.2.jar.
> > And maybe also foo-test-1.0.0.jar and foo-test-1.0.1.jar, hm...
> >
> > Thoughts?
>
> In practice the test jar is often not feasible. You will need the test
> sources, apply possible patches, compile the result and run the tests
> against that.
Yes, I expect the test jar to contain as much as possible to be helpful in debugging, including test fixtures (data files, whatever) as well as the test sources. No patches, the point is to find out what happens when you run the old tests against the current code. You want to find incompatibilities.
> This is simply because you will run sometimes in a situation
> where an old unit test verifies a wrong behavior that has been fixed in the
> new version. Therefore you need IMHO also a jar with the test sources.
>
> If you really want to use the test jar as produced by Maven as default, you
> will need a mechanism for skipping single test fixtures or declare their
> failing as expected.
Yes, I thought I outlined this here:
> > As part of the build I would run (at least) the 1.0.2 unit tests against
> > the 1.0.3 RC. If 100% pass all is well, if not, it is either a bug or a
> > known acceptable failure fixing a bug and should be documented somehow,
> > probably in a ticket.
Just to rephrase and make it clearer (in my mind at least.)
When you run the tests in an test jar, tests will pass or fail. They will fail for one of two reasons:
(1) A regression bug: The current code has changed and the behavior is no longer compatible.
(2) A bug has been fixed and that breaks old tests which expected the buggy behavior.
In the case of (1), each failure should be reported as a real problem ("red bar" in JUnit parlance)
In the case of (2), each failure should be documented and reported as a passing test.
This will require some new kind of document to track this information to be compiled by hand. Maybe something like:
<KnownTestFailures>
<FixedBug id="LANG-1234" class="FooTest" method="testFoo" />
...
Maybe this can be codified differently, I am not sure, TDB really.
Gary
>
> - Jörg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
RE: [codec][lang] Provide a test jar plus [daemon]
Posted by Jörg Schaible <jo...@gmx.de>.
Hi Gary,
Gary Gregory wrote at Monday, 5. April 2010 18:16:
> Seeing the discussion about [daemon] and not releasing made me think of
> another use for a test jar file.
>
> What I would like to know when evaluating an RC for releasing a
> maintenance of a commons component (from x.y.n to x.y.n+1) is that there
> is 100% binary compatibility.
>
> As part of the build I would run (at least) the 1.0.2 unit tests against
> the 1.0.3 RC. If 100% pass all is well, if not, it is either a bug or a
> known acceptable failure fixing a bug and should be documented somehow,
> probably in a ticket.
>
> This would mean that a release 1.0.3 RC would include foo-test-1.0.2.jar.
> And maybe also foo-test-1.0.0.jar and foo-test-1.0.1.jar, hm...
>
> Thoughts?
In practice the test jar is often not feasible. You will need the test
sources, apply possible patches, compile the result and run the tests
against that. This is simply because you will run sometimes in a situation
where an old unit test verifies a wrong behavior that has been fixed in the
new version. Therefore you need IMHO also a jar with the test sources.
If you really want to use the test jar as produced by Maven as default, you
will need a mechanism for skipping single test fixtures or declare their
failing as expected.
- Jörg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [codec][lang] Provide a test jar plus [daemon]
Posted by sebb <se...@gmail.com>.
On 05/04/2010, Gary Gregory <GG...@seagullsoftware.com> wrote:
> > -----Original Message-----
> > From: sebb [mailto:sebbaz@gmail.com]
> > Sent: Monday, April 05, 2010 11:17
> > To: Commons Developers List
> > Subject: Re: [codec][lang] Provide a test jar plus [daemon]
> >
>
> > On 05/04/2010, Gary Gregory <GG...@seagullsoftware.com> wrote:
> > > I do not think Clirr is enough as exemplified by the change of behavior in
> > Base64 encoding with CRLFs between version 1.3 and 1.4 (see CODEC-94.)
> >
> > Agreed - Clirr is necessary, but not sufficient.
> >
> > > There is no substitute for running unit tests.
> >
> > Agreed; however unit tests may not pick up changes found by Clirr.
> >
> > Ideally the tests should exercise all public APIs, but that is not
> > always possible - and rarely achieved ;-)
> >
> > > As we change code in a component, unit tests are created or changed. It is
> > possible to create compatibility problems when code and unit tests change. You
> > have to consciously and carefully leave some tests as-is from a previous
> > versions to ensure backward compatibility. Sometimes of course, you are fixing
> > bugs, so unit tests must change too.
> > >
> > > If you run the previous version's test against the current code base, you
> > are simulating what users do when they drop in a new jar in their application.
> > Since we do not have previous version's tests next to the current tests, how
> > can we test backward compatibility? If we produce a test jar file for each
> > release and keep it in SVN and deliver it, that is a snapshot of old "clients"
> > for a given version. The test phase of the build can then run all of the tests
> > in each test jar, including the test jar for the current tests, against the
> > component jar.
> > >
> >
> > I agree, it would be useful to be able to run old tests against the new code.
> > However I'm not sure that the jars should be stored in SVN. Perhaps in
> > a local snapshot repo instead?
>
>
> Where do you get the test jars initially? It would painful to have to download all releases and build your own test jars. If the point is to test backward compatibility, then we need to keep the backward bits someplace.
>
The test jars can be built as part of releases going forward.
I just don't think that they belong in SVN.
>
> >
> > If there are a lot of bug fixes in the release, then a lot of old
> > tests may fail, and it could get tedious working out which failures
> > are OK and which are not. This is not to say that the tests should not
> > be run, just that they may involve a lot of manual checking.
>
>
> Right, that would be done and documented once per release, in some XML format that can be used in a report: "Test T in class C is expected to fail because x y z." That one report would have to be carefully compiled and checked.
>
The test process might need a test POM to run the tests; perhaps
document the exceptions there?
>
> Gary
>
>
> >
> > > Gary
> > >
> > >
> > > > -----Original Message-----
> > > > From: Matt Benson [mailto:gudnabrsam@gmail.com]
> > > > Sent: Monday, April 05, 2010 10:00
> > > > To: Commons Developers List
> > >
> > > > Subject: Re: [codec][lang] Provide a test jar plus [daemon]
> > > >
> > > > Is it not sufficient to simply run clirr reports before a release?
> > > >
> > > > -Matt
> > > >
> > > > On Apr 5, 2010, at 11:16 AM, Gary Gregory wrote:
> > > >
> > > > > Seeing the discussion about [daemon] and not releasing made me
> > > > > think of another use for a test jar file.
> > > > >
> > > > > What I would like to know when evaluating an RC for releasing a
> > > > > maintenance of a commons component (from x.y.n to x.y.n+1) is that
> > > > > there is 100% binary compatibility.
> > > > >
> > > > > As part of the build I would run (at least) the 1.0.2 unit tests
> > > > > against the 1.0.3 RC. If 100% pass all is well, if not, it is
> > > > > either a bug or a known acceptable failure fixing a bug and should
> > > > > be documented somehow, probably in a ticket.
> > > > >
> > > > > This would mean that a release 1.0.3 RC would include foo-
> > > > > test-1.0.2.jar. And maybe also foo-test-1.0.0.jar and foo-
> > > > > test-1.0.1.jar, hm...
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Gary Gregory
> > > > > Senior Software Engineer
> > > > > Seagull Software
> > > > > email: ggregory@seagullsoftware.com
> > > > > email: ggregory@apache.org
> > > > > www.seagullsoftware.com
> > > > >
> > > > >
> > > > > From: Gary Gregory
> > > > > Sent: Tuesday, March 30, 2010 16:58
> > > > > To: Commons Developers List
> > > > > Subject: [codec][lang] Provide a test jar
> > > > >
> > > > > I am starting with codec and lang since it what I am most
> > > > > interested in ATM...
> > > > >
> > > > > I would like to run commons.xxx unit tests as part of my build as a
> > > > > sanity check when I try out a new combo of JVM, OS, jars, etc.
> > > > >
> > > > > Right now, I would have to compile the unit tests as part of my
> > > > > build which is not great.
> > > > >
> > > > > So, what about providing a commons-codec-1.5-test.jar file like we
> > > > > provide a sources and javadoc file?
> > > > >
> > > > > Same for any commons project really...
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Gary Gregory
> > > > > Senior Software Engineer
> > > > > Seagull Software
> > > > > email: ggregory@seagullsoftware.com
> > > > > email: ggregory@apache.org
> > > > > www.seagullsoftware.com
> > > > >
> > > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > > > For additional commands, e-mail: dev-help@commons.apache.org
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > > For additional commands, e-mail: dev-help@commons.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > For additional commands, e-mail: dev-help@commons.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
RE: [codec][lang] Provide a test jar plus [daemon]
Posted by Gary Gregory <GG...@seagullsoftware.com>.
> -----Original Message-----
> From: sebb [mailto:sebbaz@gmail.com]
> Sent: Monday, April 05, 2010 11:17
> To: Commons Developers List
> Subject: Re: [codec][lang] Provide a test jar plus [daemon]
>
> On 05/04/2010, Gary Gregory <GG...@seagullsoftware.com> wrote:
> > I do not think Clirr is enough as exemplified by the change of behavior in
> Base64 encoding with CRLFs between version 1.3 and 1.4 (see CODEC-94.)
>
> Agreed - Clirr is necessary, but not sufficient.
>
> > There is no substitute for running unit tests.
>
> Agreed; however unit tests may not pick up changes found by Clirr.
>
> Ideally the tests should exercise all public APIs, but that is not
> always possible - and rarely achieved ;-)
>
> > As we change code in a component, unit tests are created or changed. It is
> possible to create compatibility problems when code and unit tests change. You
> have to consciously and carefully leave some tests as-is from a previous
> versions to ensure backward compatibility. Sometimes of course, you are fixing
> bugs, so unit tests must change too.
> >
> > If you run the previous version's test against the current code base, you
> are simulating what users do when they drop in a new jar in their application.
> Since we do not have previous version's tests next to the current tests, how
> can we test backward compatibility? If we produce a test jar file for each
> release and keep it in SVN and deliver it, that is a snapshot of old "clients"
> for a given version. The test phase of the build can then run all of the tests
> in each test jar, including the test jar for the current tests, against the
> component jar.
> >
>
> I agree, it would be useful to be able to run old tests against the new code.
> However I'm not sure that the jars should be stored in SVN. Perhaps in
> a local snapshot repo instead?
Where do you get the test jars initially? It would painful to have to download all releases and build your own test jars. If the point is to test backward compatibility, then we need to keep the backward bits someplace.
>
> If there are a lot of bug fixes in the release, then a lot of old
> tests may fail, and it could get tedious working out which failures
> are OK and which are not. This is not to say that the tests should not
> be run, just that they may involve a lot of manual checking.
Right, that would be done and documented once per release, in some XML format that can be used in a report: "Test T in class C is expected to fail because x y z." That one report would have to be carefully compiled and checked.
Gary
>
> > Gary
> >
> >
> > > -----Original Message-----
> > > From: Matt Benson [mailto:gudnabrsam@gmail.com]
> > > Sent: Monday, April 05, 2010 10:00
> > > To: Commons Developers List
> >
> > > Subject: Re: [codec][lang] Provide a test jar plus [daemon]
> > >
> > > Is it not sufficient to simply run clirr reports before a release?
> > >
> > > -Matt
> > >
> > > On Apr 5, 2010, at 11:16 AM, Gary Gregory wrote:
> > >
> > > > Seeing the discussion about [daemon] and not releasing made me
> > > > think of another use for a test jar file.
> > > >
> > > > What I would like to know when evaluating an RC for releasing a
> > > > maintenance of a commons component (from x.y.n to x.y.n+1) is that
> > > > there is 100% binary compatibility.
> > > >
> > > > As part of the build I would run (at least) the 1.0.2 unit tests
> > > > against the 1.0.3 RC. If 100% pass all is well, if not, it is
> > > > either a bug or a known acceptable failure fixing a bug and should
> > > > be documented somehow, probably in a ticket.
> > > >
> > > > This would mean that a release 1.0.3 RC would include foo-
> > > > test-1.0.2.jar. And maybe also foo-test-1.0.0.jar and foo-
> > > > test-1.0.1.jar, hm...
> > > >
> > > > Thoughts?
> > > >
> > > > Gary Gregory
> > > > Senior Software Engineer
> > > > Seagull Software
> > > > email: ggregory@seagullsoftware.com
> > > > email: ggregory@apache.org
> > > > www.seagullsoftware.com
> > > >
> > > >
> > > > From: Gary Gregory
> > > > Sent: Tuesday, March 30, 2010 16:58
> > > > To: Commons Developers List
> > > > Subject: [codec][lang] Provide a test jar
> > > >
> > > > I am starting with codec and lang since it what I am most
> > > > interested in ATM...
> > > >
> > > > I would like to run commons.xxx unit tests as part of my build as a
> > > > sanity check when I try out a new combo of JVM, OS, jars, etc.
> > > >
> > > > Right now, I would have to compile the unit tests as part of my
> > > > build which is not great.
> > > >
> > > > So, what about providing a commons-codec-1.5-test.jar file like we
> > > > provide a sources and javadoc file?
> > > >
> > > > Same for any commons project really...
> > > >
> > > > Thoughts?
> > > >
> > > > Gary Gregory
> > > > Senior Software Engineer
> > > > Seagull Software
> > > > email: ggregory@seagullsoftware.com
> > > > email: ggregory@apache.org
> > > > www.seagullsoftware.com
> > > >
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > > For additional commands, e-mail: dev-help@commons.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > For additional commands, e-mail: dev-help@commons.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [codec][lang] Provide a test jar plus [daemon]
Posted by sebb <se...@gmail.com>.
On 05/04/2010, Gary Gregory <GG...@seagullsoftware.com> wrote:
> I do not think Clirr is enough as exemplified by the change of behavior in Base64 encoding with CRLFs between version 1.3 and 1.4 (see CODEC-94.)
Agreed - Clirr is necessary, but not sufficient.
> There is no substitute for running unit tests.
Agreed; however unit tests may not pick up changes found by Clirr.
Ideally the tests should exercise all public APIs, but that is not
always possible - and rarely achieved ;-)
> As we change code in a component, unit tests are created or changed. It is possible to create compatibility problems when code and unit tests change. You have to consciously and carefully leave some tests as-is from a previous versions to ensure backward compatibility. Sometimes of course, you are fixing bugs, so unit tests must change too.
>
> If you run the previous version's test against the current code base, you are simulating what users do when they drop in a new jar in their application. Since we do not have previous version's tests next to the current tests, how can we test backward compatibility? If we produce a test jar file for each release and keep it in SVN and deliver it, that is a snapshot of old "clients" for a given version. The test phase of the build can then run all of the tests in each test jar, including the test jar for the current tests, against the component jar.
>
I agree, it would be useful to be able to run old tests against the new code.
However I'm not sure that the jars should be stored in SVN. Perhaps in
a local snapshot repo instead?
If there are a lot of bug fixes in the release, then a lot of old
tests may fail, and it could get tedious working out which failures
are OK and which are not. This is not to say that the tests should not
be run, just that they may involve a lot of manual checking.
> Gary
>
>
> > -----Original Message-----
> > From: Matt Benson [mailto:gudnabrsam@gmail.com]
> > Sent: Monday, April 05, 2010 10:00
> > To: Commons Developers List
>
> > Subject: Re: [codec][lang] Provide a test jar plus [daemon]
> >
> > Is it not sufficient to simply run clirr reports before a release?
> >
> > -Matt
> >
> > On Apr 5, 2010, at 11:16 AM, Gary Gregory wrote:
> >
> > > Seeing the discussion about [daemon] and not releasing made me
> > > think of another use for a test jar file.
> > >
> > > What I would like to know when evaluating an RC for releasing a
> > > maintenance of a commons component (from x.y.n to x.y.n+1) is that
> > > there is 100% binary compatibility.
> > >
> > > As part of the build I would run (at least) the 1.0.2 unit tests
> > > against the 1.0.3 RC. If 100% pass all is well, if not, it is
> > > either a bug or a known acceptable failure fixing a bug and should
> > > be documented somehow, probably in a ticket.
> > >
> > > This would mean that a release 1.0.3 RC would include foo-
> > > test-1.0.2.jar. And maybe also foo-test-1.0.0.jar and foo-
> > > test-1.0.1.jar, hm...
> > >
> > > Thoughts?
> > >
> > > Gary Gregory
> > > Senior Software Engineer
> > > Seagull Software
> > > email: ggregory@seagullsoftware.com
> > > email: ggregory@apache.org
> > > www.seagullsoftware.com
> > >
> > >
> > > From: Gary Gregory
> > > Sent: Tuesday, March 30, 2010 16:58
> > > To: Commons Developers List
> > > Subject: [codec][lang] Provide a test jar
> > >
> > > I am starting with codec and lang since it what I am most
> > > interested in ATM...
> > >
> > > I would like to run commons.xxx unit tests as part of my build as a
> > > sanity check when I try out a new combo of JVM, OS, jars, etc.
> > >
> > > Right now, I would have to compile the unit tests as part of my
> > > build which is not great.
> > >
> > > So, what about providing a commons-codec-1.5-test.jar file like we
> > > provide a sources and javadoc file?
> > >
> > > Same for any commons project really...
> > >
> > > Thoughts?
> > >
> > > Gary Gregory
> > > Senior Software Engineer
> > > Seagull Software
> > > email: ggregory@seagullsoftware.com
> > > email: ggregory@apache.org
> > > www.seagullsoftware.com
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > For additional commands, e-mail: dev-help@commons.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
RE: [codec][lang] Provide a test jar plus [daemon]
Posted by Gary Gregory <GG...@seagullsoftware.com>.
I do not think Clirr is enough as exemplified by the change of behavior in Base64 encoding with CRLFs between version 1.3 and 1.4 (see CODEC-94.)
There is no substitute for running unit tests.
As we change code in a component, unit tests are created or changed. It is possible to create compatibility problems when code and unit tests change. You have to consciously and carefully leave some tests as-is from a previous versions to ensure backward compatibility. Sometimes of course, you are fixing bugs, so unit tests must change too.
If you run the previous version's test against the current code base, you are simulating what users do when they drop in a new jar in their application. Since we do not have previous version's tests next to the current tests, how can we test backward compatibility? If we produce a test jar file for each release and keep it in SVN and deliver it, that is a snapshot of old "clients" for a given version. The test phase of the build can then run all of the tests in each test jar, including the test jar for the current tests, against the component jar.
Gary
> -----Original Message-----
> From: Matt Benson [mailto:gudnabrsam@gmail.com]
> Sent: Monday, April 05, 2010 10:00
> To: Commons Developers List
> Subject: Re: [codec][lang] Provide a test jar plus [daemon]
>
> Is it not sufficient to simply run clirr reports before a release?
>
> -Matt
>
> On Apr 5, 2010, at 11:16 AM, Gary Gregory wrote:
>
> > Seeing the discussion about [daemon] and not releasing made me
> > think of another use for a test jar file.
> >
> > What I would like to know when evaluating an RC for releasing a
> > maintenance of a commons component (from x.y.n to x.y.n+1) is that
> > there is 100% binary compatibility.
> >
> > As part of the build I would run (at least) the 1.0.2 unit tests
> > against the 1.0.3 RC. If 100% pass all is well, if not, it is
> > either a bug or a known acceptable failure fixing a bug and should
> > be documented somehow, probably in a ticket.
> >
> > This would mean that a release 1.0.3 RC would include foo-
> > test-1.0.2.jar. And maybe also foo-test-1.0.0.jar and foo-
> > test-1.0.1.jar, hm...
> >
> > Thoughts?
> >
> > Gary Gregory
> > Senior Software Engineer
> > Seagull Software
> > email: ggregory@seagullsoftware.com
> > email: ggregory@apache.org
> > www.seagullsoftware.com
> >
> >
> > From: Gary Gregory
> > Sent: Tuesday, March 30, 2010 16:58
> > To: Commons Developers List
> > Subject: [codec][lang] Provide a test jar
> >
> > I am starting with codec and lang since it what I am most
> > interested in ATM...
> >
> > I would like to run commons.xxx unit tests as part of my build as a
> > sanity check when I try out a new combo of JVM, OS, jars, etc.
> >
> > Right now, I would have to compile the unit tests as part of my
> > build which is not great.
> >
> > So, what about providing a commons-codec-1.5-test.jar file like we
> > provide a sources and javadoc file?
> >
> > Same for any commons project really...
> >
> > Thoughts?
> >
> > Gary Gregory
> > Senior Software Engineer
> > Seagull Software
> > email: ggregory@seagullsoftware.com
> > email: ggregory@apache.org
> > www.seagullsoftware.com
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [codec][lang] Provide a test jar plus [daemon]
Posted by Matt Benson <gu...@gmail.com>.
Is it not sufficient to simply run clirr reports before a release?
-Matt
On Apr 5, 2010, at 11:16 AM, Gary Gregory wrote:
> Seeing the discussion about [daemon] and not releasing made me
> think of another use for a test jar file.
>
> What I would like to know when evaluating an RC for releasing a
> maintenance of a commons component (from x.y.n to x.y.n+1) is that
> there is 100% binary compatibility.
>
> As part of the build I would run (at least) the 1.0.2 unit tests
> against the 1.0.3 RC. If 100% pass all is well, if not, it is
> either a bug or a known acceptable failure fixing a bug and should
> be documented somehow, probably in a ticket.
>
> This would mean that a release 1.0.3 RC would include foo-
> test-1.0.2.jar. And maybe also foo-test-1.0.0.jar and foo-
> test-1.0.1.jar, hm...
>
> Thoughts?
>
> Gary Gregory
> Senior Software Engineer
> Seagull Software
> email: ggregory@seagullsoftware.com
> email: ggregory@apache.org
> www.seagullsoftware.com
>
>
> From: Gary Gregory
> Sent: Tuesday, March 30, 2010 16:58
> To: Commons Developers List
> Subject: [codec][lang] Provide a test jar
>
> I am starting with codec and lang since it what I am most
> interested in ATM...
>
> I would like to run commons.xxx unit tests as part of my build as a
> sanity check when I try out a new combo of JVM, OS, jars, etc.
>
> Right now, I would have to compile the unit tests as part of my
> build which is not great.
>
> So, what about providing a commons-codec-1.5-test.jar file like we
> provide a sources and javadoc file?
>
> Same for any commons project really...
>
> Thoughts?
>
> Gary Gregory
> Senior Software Engineer
> Seagull Software
> email: ggregory@seagullsoftware.com
> email: ggregory@apache.org
> www.seagullsoftware.com
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
RE: [codec][lang] Provide a test jar plus [daemon]
Posted by Gary Gregory <GG...@seagullsoftware.com>.
For [lang] for example, we would have commons-lang-test-1.0.0.jar through commons-lang-test-2.5.0.jar available when a build is run.
The files should really be put up wherever we stash the other deliverables. Since 2.5 is out we cannot (?) add these jar files to it but perhaps this could be done for a 2.5.1 and for certain for 3.0. In the case of [lang] 3.0, since we changed the package name, the tests would not run, but you get the idea. Again, for [lang] we could take the 2.5 tests, clone them and only change package references to point to lang3. That would be a one-time deal when a package name changes.
For each test jar, you run all *Test* classes in it (or some suitable configurable pattern) against the current [foo] jar. For future releases, this will be easier using JUnit 4 since JUnit should pick up anything marked with @Test.
The test reports will then need to be gathered and added to the reports list. We should see the same Surefire summary and details page like we do now for the current tests, but see one report for each binary compatibility test jar. Can Maven handle that?
Gary
> -----Original Message-----
> From: Henri Yandell [mailto:flamefew@gmail.com]
> Sent: Monday, April 05, 2010 21:34
> To: Commons Developers List
> Subject: Re: [codec][lang] Provide a test jar plus [daemon]
>
> Would love to see this.
>
> I'll go and put the historic Lang ones together if you let me know how
> you'd like it to look.
>
> Hen
>
> On Mon, Apr 5, 2010 at 9:16 AM, Gary Gregory
> <GG...@seagullsoftware.com> wrote:
> > Seeing the discussion about [daemon] and not releasing made me think of
> another use for a test jar file.
> >
> > What I would like to know when evaluating an RC for releasing a maintenance
> of a commons component (from x.y.n to x.y.n+1) is that there is 100% binary
> compatibility.
> >
> > As part of the build I would run (at least) the 1.0.2 unit tests against the
> 1.0.3 RC. If 100% pass all is well, if not, it is either a bug or a known
> acceptable failure fixing a bug and should be documented somehow, probably in
> a ticket.
> >
> > This would mean that a release 1.0.3 RC would include foo-test-1.0.2.jar.
> And maybe also foo-test-1.0.0.jar and foo-test-1.0.1.jar, hm...
> >
> > Thoughts?
> >
> > Gary Gregory
> > Senior Software Engineer
> > Seagull Software
> > email: ggregory@seagullsoftware.com
> > email: ggregory@apache.org
> > www.seagullsoftware.com
> >
> >
> > From: Gary Gregory
> > Sent: Tuesday, March 30, 2010 16:58
> > To: Commons Developers List
> > Subject: [codec][lang] Provide a test jar
> >
> > I am starting with codec and lang since it what I am most interested in
> ATM...
> >
> > I would like to run commons.xxx unit tests as part of my build as a sanity
> check when I try out a new combo of JVM, OS, jars, etc.
> >
> > Right now, I would have to compile the unit tests as part of my build which
> is not great.
> >
> > So, what about providing a commons-codec-1.5-test.jar file like we provide a
> sources and javadoc file?
> >
> > Same for any commons project really...
> >
> > Thoughts?
> >
> > Gary Gregory
> > Senior Software Engineer
> > Seagull Software
> > email: ggregory@seagullsoftware.com
> > email: ggregory@apache.org
> > www.seagullsoftware.com
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [codec][lang] Provide a test jar plus [daemon]
Posted by Henri Yandell <fl...@gmail.com>.
Would love to see this.
I'll go and put the historic Lang ones together if you let me know how
you'd like it to look.
Hen
On Mon, Apr 5, 2010 at 9:16 AM, Gary Gregory
<GG...@seagullsoftware.com> wrote:
> Seeing the discussion about [daemon] and not releasing made me think of another use for a test jar file.
>
> What I would like to know when evaluating an RC for releasing a maintenance of a commons component (from x.y.n to x.y.n+1) is that there is 100% binary compatibility.
>
> As part of the build I would run (at least) the 1.0.2 unit tests against the 1.0.3 RC. If 100% pass all is well, if not, it is either a bug or a known acceptable failure fixing a bug and should be documented somehow, probably in a ticket.
>
> This would mean that a release 1.0.3 RC would include foo-test-1.0.2.jar. And maybe also foo-test-1.0.0.jar and foo-test-1.0.1.jar, hm...
>
> Thoughts?
>
> Gary Gregory
> Senior Software Engineer
> Seagull Software
> email: ggregory@seagullsoftware.com
> email: ggregory@apache.org
> www.seagullsoftware.com
>
>
> From: Gary Gregory
> Sent: Tuesday, March 30, 2010 16:58
> To: Commons Developers List
> Subject: [codec][lang] Provide a test jar
>
> I am starting with codec and lang since it what I am most interested in ATM...
>
> I would like to run commons.xxx unit tests as part of my build as a sanity check when I try out a new combo of JVM, OS, jars, etc.
>
> Right now, I would have to compile the unit tests as part of my build which is not great.
>
> So, what about providing a commons-codec-1.5-test.jar file like we provide a sources and javadoc file?
>
> Same for any commons project really...
>
> Thoughts?
>
> Gary Gregory
> Senior Software Engineer
> Seagull Software
> email: ggregory@seagullsoftware.com
> email: ggregory@apache.org
> www.seagullsoftware.com
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org