[Heads up][code] Apache Lucene updated to version 2.9.4

[Pedro Giffuni <pf...@apache.org>]

Hello;

Revision 1337602 updates Apache Lucene to version 2.9.4. It's the
last update for Apache based packages that I have planned. It
also brings in a patch so that more modern versions can be used
as external libraries.

The change is rather big and I have a lot of unrelated patches in
my build so I may have accidentally left something in/out. Please
do report build issues and be patient :).

The change works well on UNIX, but it also involved getting rid
of a hack (long_path.patch) in Lucene that we carried for
Windows. I am unsure if the change is still necessary so testing
the help on Windows is encouraged.

This change will *not* be merged back to 3.4.1.

Pedro.

Re: [Heads up][code] Apache Lucene updated to version 2.9.4

[Ariel Constenla-Haile <ar...@apache.org>]

Hi Rob, Pedro,

On Sun, May 13, 2012 at 09:57:55AM -0400, Rob Weir wrote:
> On Sat, May 12, 2012 at 9:24 PM, Pedro Giffuni <pf...@apache.org> wrote:
> > Hi Ariel;
> >
> > On 05/12/12 16:10, Ariel Constenla-Haile wrote:
> >>
> >> Hi Pedro,
> >> ...
> >>
> >> IMO when updating external dependencies, the tests should not only include
> >> the fact that it can be built on all the platforms we support, but
> >> mainly regressions tests that test if the functionality of the code that
> >> dependes on these external dependencies is still working.
> >>
> >> The steps would be:
> >>
> >> 1. make sure it builds
> >> 2. identify the code that depends on the dependency
> >> 3. test that the functionality still works.
> >>
> >> This is valid also for the apache commons update you did recently.
> >
> >
> > With Java code there are always two particular things:
> > 1) In theory Java code is platform independent, so even though
> > I can't test every platform, having it work in UNIX is a really
> > good sign.
> > 2) One might think the usual "if it ain't broken don't fix it"
> > philosophy will keep things stable ... but it doesn't: even if we
> > would like to keep the actual working code set in stone, the
> > Java VM does get updated and previously working code stops
> > working or doesn't even build anymore.
> >
> > Andrew Rist reported a broken case of a linux buildbot with
> > openjdk7, and he requested the Apache commons update.
> >
> > Indeed, as you are aware, I am not perfect and you were hit
> > with many of my early commits that caused breakage (even
> > when I had all of them reviewed by someone else that knew
> > the code better than me).
> >
> 
> Whew!  I thought I was the only non-perfect person here ;-)
> 
> But seriously, no large development effort can ever rely on perfect
> (or near-perfect) developers.  That approach doesn't scale.   We need
> to rely on an overall process that can efficiently find errors, and
> find them early.
> 
> So I wonder, in cases like this, where we're upgrading a library that
> might cause functional regressions, whether we should do something
> like this:
> 
> 1) Open a BZ issue for the task, e.g., upgrading a particular library.
> 
> 2) In the issue, describe the general functionality that may be
> effected by the library upgrade
> 
> 3) This then gives the QA volunteers a "head's up" that they should do
> some deeper testing in this area.  (They probably don't read every
> message on ooo-commits)
> 
> 4) It also gives us a place where we can look for producing release
> notes for 3.5.
> 
> Does this make sense?

You expressed it more clearly than me :)


Regards
-- 
Ariel Constenla-Haile
La Plata, Argentina

Re: [Heads up][code] Apache Lucene updated to version 2.9.4

[Pedro Giffuni <pf...@apache.org>]

On 05/13/12 08:57, Rob Weir wrote:
> ...
> Whew!  I thought I was the only non-perfect person here ;-)

:)

> But seriously, no large development effort can ever rely on perfect
> (or near-perfect) developers.  That approach doesn't scale.   We need
> to rely on an overall process that can efficiently find errors, and
> find them early.
>
> So I wonder, in cases like this, where we're upgrading a library that
> might cause functional regressions, whether we should do something
> like this:
>
> 1) Open a BZ issue for the task, e.g., upgrading a particular library.
>
> 2) In the issue, describe the general functionality that may be
> effected by the library upgrade
>
> 3) This then gives the QA volunteers a "head's up" that they should do
> some deeper testing in this area.  (They probably don't read every
> message on ooo-commits)
>
> 4) It also gives us a place where we can look for producing release
> notes for 3.5.
>
> Does this make sense?
>

Makes perfect sense.

In this case I reused BZ issue 115241 which already existed for
the Lucene update. I didn't note it on the top as there were more
changes going on. After committing the change (and this time
the log was rather descriptive of the change) I added a
reference to the commit as I always do.

The issue, I think, is not really what to do but how to improve
the workflow: I set the bug to RESOLVED FIXED. Is this the
correct way to give a heads up for QA?

Pedro.

Re: [Heads up][code] Apache Lucene updated to version 2.9.4

[Rob Weir <ro...@apache.org>]

On Sat, May 12, 2012 at 9:24 PM, Pedro Giffuni <pf...@apache.org> wrote:
> Hi Ariel;
>
> On 05/12/12 16:10, Ariel Constenla-Haile wrote:
>>
>> Hi Pedro,
>> ...
>>
>> IMO when updating external dependencies, the tests should not only include
>> the fact that it can be built on all the platforms we support, but
>> mainly regressions tests that test if the functionality of the code that
>> dependes on these external dependencies is still working.
>>
>> The steps would be:
>>
>> 1. make sure it builds
>> 2. identify the code that depends on the dependency
>> 3. test that the functionality still works.
>>
>> This is valid also for the apache commons update you did recently.
>
>
> With Java code there are always two particular things:
> 1) In theory Java code is platform independent, so even though
> I can't test every platform, having it work in UNIX is a really
> good sign.
> 2) One might think the usual "if it ain't broken don't fix it"
> philosophy will keep things stable ... but it doesn't: even if we
> would like to keep the actual working code set in stone, the
> Java VM does get updated and previously working code stops
> working or doesn't even build anymore.
>
> Andrew Rist reported a broken case of a linux buildbot with
> openjdk7, and he requested the Apache commons update.
>
> Indeed, as you are aware, I am not perfect and you were hit
> with many of my early commits that caused breakage (even
> when I had all of them reviewed by someone else that knew
> the code better than me).
>

Whew!  I thought I was the only non-perfect person here ;-)

But seriously, no large development effort can ever rely on perfect
(or near-perfect) developers.  That approach doesn't scale.   We need
to rely on an overall process that can efficiently find errors, and
find them early.

So I wonder, in cases like this, where we're upgrading a library that
might cause functional regressions, whether we should do something
like this:

1) Open a BZ issue for the task, e.g., upgrading a particular library.

2) In the issue, describe the general functionality that may be
effected by the library upgrade

3) This then gives the QA volunteers a "head's up" that they should do
some deeper testing in this area.  (They probably don't read every
message on ooo-commits)

4) It also gives us a place where we can look for producing release
notes for 3.5.

Does this make sense?

-Rob


> To respond your points, which are perfectly reasonable, in the
> case of these last two big changes:
>
> 1) The code builds on FreeBSD-amd64, which is my dev. platform.
> 2) The code has been in use for a while on FreeBSD and/or Debian
> Linux for a while. I was very careful to choose only compatible
> updates.
> 3) I did my best to check the specific functionality that may be
> affected: it is unlikely I can catch all the use-cases but doing
> such changes early in development will help detect any
> remaining issue.
>
> This said, we are reaching a level where updating this low-hanging
> fruit is not becoming viable anymore. Updating commons-lang to
> version 3.1 or Lucene to 3.5 would involve a *lot* of work but I
> guess it will get done when it's mandatory to use Java5+.
>
> I know my limitations and I can't really help beyond the updates
> I am doing but I am sure this is pretty useful for now.
>
> Pedro.
>

Re: [Heads up][code] Apache Lucene updated to version 2.9.4

[Pedro Giffuni <pf...@apache.org>]

On 05/13/12 09:10, Ariel Constenla-Haile wrote:
> Hi Pedro,
>
> On Sat, May 12, 2012 at 08:24:00PM -0500, Pedro Giffuni wrote:
>> To respond your points, which are perfectly reasonable, in the
>> case of these last two big changes:
>>
>> 1) The code builds on FreeBSD-amd64, which is my dev. platform.
>> 2) The code has been in use for a while on FreeBSD and/or Debian
>> Linux for a while. I was very careful to choose only compatible
>> updates.
>> 3) I did my best to check the specific functionality that may be
>> affected: it is unlikely I can catch all the use-cases but doing
>> such changes early in development will help detect any
>> remaining issue.
> We have QA experts here ;) The idea behind my mail was: identify where
> the libraries you are updating is used, then ask the QA people if they
> can perform some tests (or ask here in the list for testing volunteers).
> For people to test functionality, you must tell them first where/what to
> test.

Ahh.. OK.. so we are in the same "channel", because that was the
reason for the [heads up] tag :). I do want the Lucene change tested
on Windows: specifically for looking up long paths in the help system.

The old port had an awful hack that requires an effort to update: in
general adding system independent hacks in Java code is not good
so I want to know if that extra effort is needed at all.

> For example, with apache commons, I have no idea where this can be used,
> opengrok suggests in the Report Builder (we don't build it anymore) and
> the Wiki Publisher (this extension is not installed by default with the
> office, and AFAIK we didn't upload a new version on the extensions
> repository).

Those are used for logging and http access in extensions, however,
I am not worried about Apache Commons. I only updated lang
and codec which are used by http-client: the updated versions have
been in use in FreeBSD's ports for a long time and I verified the
changelogs of both updated to make sure there were no surprises
like deprecated APIs.

> In conclusion, for the third step, we should identify where the code is
> used, ask for QA volunteers telling them what to test.

Hm ... I guess I will have to join the QA list in addition to the heads-up.

Pedro.

Re: [Heads up][code] Apache Lucene updated to version 2.9.4

[Ariel Constenla-Haile <ar...@apache.org>]

Hi Pedro,

On Sat, May 12, 2012 at 08:24:00PM -0500, Pedro Giffuni wrote:
> To respond your points, which are perfectly reasonable, in the
> case of these last two big changes:
> 
> 1) The code builds on FreeBSD-amd64, which is my dev. platform.
> 2) The code has been in use for a while on FreeBSD and/or Debian
> Linux for a while. I was very careful to choose only compatible
> updates.
> 3) I did my best to check the specific functionality that may be
> affected: it is unlikely I can catch all the use-cases but doing
> such changes early in development will help detect any
> remaining issue.

We have QA experts here ;) The idea behind my mail was: identify where
the libraries you are updating is used, then ask the QA people if they
can perform some tests (or ask here in the list for testing volunteers).
For people to test functionality, you must tell them first where/what to
test.

For example, with apache commons, I have no idea where this can be used,
opengrok suggests in the Report Builder (we don't build it anymore) and
the Wiki Publisher (this extension is not installed by default with the
office, and AFAIK we didn't upload a new version on the extensions
repository). 

In conclusion, for the third step, we should identify where the code is
used, ask for QA volunteers telling them what to test.


Regards
-- 
Ariel Constenla-Haile
La Plata, Argentina

Re: [Heads up][code] Apache Lucene updated to version 2.9.4

[Pedro Giffuni <pf...@apache.org>]

Hi Ariel;

On 05/12/12 16:10, Ariel Constenla-Haile wrote:
> Hi Pedro,
> ...
> IMO when updating external dependencies, the tests should not only include
> the fact that it can be built on all the platforms we support, but
> mainly regressions tests that test if the functionality of the code that
> dependes on these external dependencies is still working.
>
> The steps would be:
>
> 1. make sure it builds
> 2. identify the code that depends on the dependency
> 3. test that the functionality still works.
>
> This is valid also for the apache commons update you did recently.

With Java code there are always two particular things:
1) In theory Java code is platform independent, so even though
I can't test every platform, having it work in UNIX is a really
good sign.
2) One might think the usual "if it ain't broken don't fix it"
philosophy will keep things stable ... but it doesn't: even if we
would like to keep the actual working code set in stone, the
Java VM does get updated and previously working code stops
working or doesn't even build anymore.

Andrew Rist reported a broken case of a linux buildbot with
openjdk7, and he requested the Apache commons update.

Indeed, as you are aware, I am not perfect and you were hit
with many of my early commits that caused breakage (even
when I had all of them reviewed by someone else that knew
the code better than me).

To respond your points, which are perfectly reasonable, in the
case of these last two big changes:

1) The code builds on FreeBSD-amd64, which is my dev. platform.
2) The code has been in use for a while on FreeBSD and/or Debian
Linux for a while. I was very careful to choose only compatible
updates.
3) I did my best to check the specific functionality that may be
affected: it is unlikely I can catch all the use-cases but doing
such changes early in development will help detect any
remaining issue.

This said, we are reaching a level where updating this low-hanging
fruit is not becoming viable anymore. Updating commons-lang to
version 3.1 or Lucene to 3.5 would involve a *lot* of work but I
guess it will get done when it's mandatory to use Java5+.

I know my limitations and I can't really help beyond the updates
I am doing but I am sure this is pretty useful for now.

Pedro.

Re: [Heads up][code] Apache Lucene updated to version 2.9.4

[Ariel Constenla-Haile <ar...@apache.org>]

Hi Pedro,

On Sat, May 12, 2012 at 01:24:01PM -0500, Pedro Giffuni wrote:
> Hello;
> 
> Revision 1337602 updates Apache Lucene to version 2.9.4. It's the
> last update for Apache based packages that I have planned. It
> also brings in a patch so that more modern versions can be used
> as external libraries.
> 
> The change is rather big and I have a lot of unrelated patches in
> my build so I may have accidentally left something in/out. Please
> do report build issues and be patient :).
> 
> The change works well on UNIX, but it also involved getting rid
> of a hack (long_path.patch) in Lucene that we carried for
> Windows. I am unsure if the change is still necessary so testing
> the help on Windows is encouraged.

IMO when updating external dependencies, the tests should not only include
the fact that it can be built on all the platforms we support, but
mainly regressions tests that test if the functionality of the code that
dependes on these external dependencies is still working.

The steps would be:

1. make sure it builds
2. identify the code that depends on the dependency
3. test that the functionality still works.

This is valid also for the apache commons update you did recently.


Regards
-- 
Ariel Constenla-Haile
La Plata, Argentina