What are we allowed to do in 3.0?

[Shai Erera <se...@gmail.com>]

Hi

Recently I was involved in several issues that required some runtime changes
to be done in 3.0 and it was not so clear what is it that we're actually
allowed to do in 3.0. So I'd like to open it for discussion, unless
everybody agree on it already.

So far I can tell that 3.0 allows us to:
1) Get rid of all the deprecated methods
2) Move to Java 1.5

But what about changes to runtime behavior, re-factoring a whole set of
classes etc? I'd like to relate to them one by one, so that we can comment
on each.

Removing deprecated methods
---------------------------------------------
As I was told, 2.9 is the last release we are allowed to mark methods as
deprecated, and remove them in 3.0. I.e., after 2.9 is out, if we feel there
is a method that should be renamed, its signature should change or be
removed altogether, we can't just do it and we'd have to deprecate it and
remove it in 4.0 (?). I personally thought that 2.9 allows us to make these
changes without letting anyone know about them in advance, which I'm ok with
since upgrading to 3.0 is not going to be as 'smooth', but I also understand
why 'letting people know in advance' (which means a release prior to the one
when they are removed) gives a "better service" to our users. I also thought
that jar drop-in-ability is not supposed to be supported from 2.9 to 3.0
(but I was corrected previously on that).
Which means upon releasing 2.9, the very first issue that's opened and
committed should be removing the current deprecated methods, or otherwise we
could open an issue that deprecates a method and accidentally remove it
later, when we handle the massive deprecation removal. We should also create
a 2.9 tag.

Changes to runtime behavior
-----------------------------------------
What is exactly the policy here? If we document in 2.9 that certain
features' runtime behavior will change in 3.0 - is that ok to make those
changes? And if we don't document them and do it (in the transition from
2.9-3.0) then it's not? Why? After all, I expect anyone who upgrades to 3.0
to run all of his unit tests to assert that everything still works (I expect
that to happen for every release, but for 3.0 in particular). Obviously the
runtime behaviors that were changed and documented in 2.9 are ones that he
might have already taken care of, but why can't he do the same reading the
CHANGES of 3.0?
I just feel that this policy forces us to think really hard and foresee
those changes in runtime behavior that we'd like to do in 3.0 so that we can
get them into 2.9, but at the end of the day we're not improving anything.
Upon upgrading to 2.9 I cannot handle the changes in runtime behaviors as
they weren't done yet. I can only do it after I upgrade to 3.0. So what's
the difference for me between fixing those that were documented in 2.9 and
the new ones that were just released?

Going forward, I don't think this community changes runtime behavior every
other Monday, and so I'd like to have the ability to make those changes
without such a strict policy. Those changes are meant to help our users (and
we are amongst them) to achieve better performance, usually, and so why
should we fear from making them, or if fear is too strong a word - why
should we refrain from doing them, while documenting the changes? If we
don't want to do it for every 'dot' release, we can do them in major
releases and I'd also vote for doing them in a mid-major release, like 3.5.

Refactoring
----------------
Today we are quite limited with refactoring. We cannot add methods to
interfaces or abstract methods to abstract classes, or even make classes
abstract. I'm perfectly fine with that as I don't want to face the need to
suddenly refactor my application just because Lucene decided to add a method
to an interface.

But recently, while working on LUCENE-1593, me and Mike spotted a need to
add some methods to Weight, but since it is an interface we can't. So I said
something "let's do it in 3.0" but then we were not sure if this can be done
in 3.0. So the alternative was "let's deprecate Weight, create an
AbstractWeight class and do it there", but we weren't sure if that's even
something we can push for in 3.0, unless we do all of it in 2.9. This also
messes up the code, introducing new classes with bad names (AbstractWeight,
AbstractSearchable) where we could have avoided it if we just changed Weight
to an abstract class in 3.0.

---------------------------

I think it all boils down to whether we MUST support jar drop-in-ability
when upgrading from 2.9 to 3.0. I think that we shouldn't as the whole
notion of 3.0 (or any future major version) is major revision to code, index
structure, JDK etc. If we're always expected to support it, then 2.9 really
becomes 3.0 in terms of our ability to make changes to the API between
2.9-3.0. I'm afraid that if that's the case, we might choose to hold on with
2.9 as much as we can so we can push as many changes as we foresee into it,
so that they can be finalized in 3.0. I'm not sure what will that give us,
because upong upgrading to 2.9 I'll have to make a major revision to my code
if I want to simply drop the 3.0 jar in.

And while back-compat of index structure is a MUST (i.e., I don't want to
rebuild my indexes every time I upgrade a major version), doing some house
cleaning to my code and taking advantage of the new classes and/or more
performing API is something I'm willing to live with.

If the answer to all my questions is "we MUST do it this way because that's
what we've decided long ago", then so be it. But if we're willing to
re-consider our back-compat policy and maybe even change it, then it's
something we can put in the CHANGES of 2.9, going forward to 3.0 :)

Shai

Re: What are we allowed to do in 3.0?

[Shai Erera <se...@gmail.com>]

So I understand from your responses that you think that 2.9 should include
as much as possible, so that a user will have ~0 work upgrading from 2.9 to
3.0, assuming he upgraded fully to 2.9 (moved to use the not-deprecated APIs
etc.).
If 3.0 is supposed to be released quickly after 2.9 then this makes sense,
and leaves very small room for sudden major changes anyway.

Also I read that you do support introducing minor changes to the API as well
as runtime behvavior, but still prefer that we do them in 2.9. So
refactoring like we discussed - changing all interfaces to abstract classes
- should not happen in 2.9->3.0, which makes sense. I think anyway this type
of refactoring should happen "one at a time", and not a complete overhaul of
the code.

Well ... I guess that if that's the case (releasing 3.0 soon after we
release 2.9, and introducing very minor and few changes to API and runtime
behavior that were not documented in 2.9 may be acceptable), then we should
be fine and this very short discussion (I somehow expected much more
responses) can end.

Thanks a lot for clarifying that to me.

Shai

On Thu, Apr 30, 2009 at 6:20 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> I think 3.0 should be a fast turnaround after 2.9?  Ie, no "new
> development" should take place?  We should remove deprecated APIs,
> change defaults, etc., but that's about it.  (I think this is how past
> major releases worked?).  It's a "fast switch".  Which then means we
> need to do all the hard work in 2.9...
>
> So, I think any API changes we want to make must be present in 2.9 as
> deprecations.  We shouldn't up and remove / rename something in 3.0
> with no fore-warning in 2.9.  Is there a case where this is too
> painful?
>
> Likewise I think we should give notification of expected changes in
> runtime behavior with 2.9 (and not suddenly do them in 3.0).
>
> > Which means upon releasing 2.9, the very first issue that's opened and
> committed should be removing the current deprecated methods, or otherwise we
> could open an issue that deprecates a method and accidentally remove it
> later, when we handle the massive deprecation removal.
>
> I think we should not target JAR drop-inability, and we should allow
> changes to runtime behavior, as well as certain minor API changes in
> 3.0.  EG here are some of the changes already slated for 3.0:
>
>  * IndexReader.open returns readOnly reader by default
>
>  * IndexReader.norms returns null on fields that don't have norms
>
>  * InterruptedException is thrown by many APIs
>
>  * IndexWriter.autoCommit is hardwired to false
>
>  * Things that now return the deprecated IndexCommitPoint (interface)
>    will be changed to return IndexCommit (abstract base class)
>
>  * Directory.list will be removed; Directory.listAll will become an
>    abstract method
>
>  * Stop tracking scores by default when sorting by field
>
> > But recently, while working on LUCENE-1593, me and Mike spotted a need to
> add some methods to Weight, but since it is an interface we can't. So I said
> something "let's do it in 3.0" but then we were not sure if this can be done
> in 3.0.
>
> I think for this we should probably introduce an abstract base class
> (implementing Weight) in 2.9, stating that Weight interface will be
> removed in 3.0?  (EG, this is what was done for
> IndexCommitPoint/IndexCommit).  Simply changing Weight to be an
> abstract class in 3.0 is spooky because Java is single inheritance, ie
> for existing classes that "implements Weight" but subclass something
> else it would be nicer to give a heads up with 2.9 that they'll need
> to refactor?
>
> Mike
>
> On Thu, Apr 30, 2009 at 6:25 AM, Shai Erera <se...@gmail.com> wrote:
> > Hi
> >
> > Recently I was involved in several issues that required some runtime
> changes
> > to be done in 3.0 and it was not so clear what is it that we're actually
> > allowed to do in 3.0. So I'd like to open it for discussion, unless
> > everybody agree on it already.
> >
> > So far I can tell that 3.0 allows us to:
> > 1) Get rid of all the deprecated methods
> > 2) Move to Java 1.5
> >
> > But what about changes to runtime behavior, re-factoring a whole set of
> > classes etc? I'd like to relate to them one by one, so that we can
> comment
> > on each.
> >
> > Removing deprecated methods
> > ---------------------------------------------
> > As I was told, 2.9 is the last release we are allowed to mark methods as
> > deprecated, and remove them in 3.0. I.e., after 2.9 is out, if we feel
> there
> > is a method that should be renamed, its signature should change or be
> > removed altogether, we can't just do it and we'd have to deprecate it and
> > remove it in 4.0 (?). I personally thought that 2.9 allows us to make
> these
> > changes without letting anyone know about them in advance, which I'm ok
> with
> > since upgrading to 3.0 is not going to be as 'smooth', but I also
> understand
> > why 'letting people know in advance' (which means a release prior to the
> one
> > when they are removed) gives a "better service" to our users. I also
> thought
> > that jar drop-in-ability is not supposed to be supported from 2.9 to 3.0
> > (but I was corrected previously on that).
> > Which means upon releasing 2.9, the very first issue that's opened and
> > committed should be removing the current deprecated methods, or otherwise
> we
> > could open an issue that deprecates a method and accidentally remove it
> > later, when we handle the massive deprecation removal. We should also
> create
> > a 2.9 tag.
> >
> > Changes to runtime behavior
> > -----------------------------------------
> > What is exactly the policy here? If we document in 2.9 that certain
> > features' runtime behavior will change in 3.0 - is that ok to make those
> > changes? And if we don't document them and do it (in the transition from
> > 2.9-3.0) then it's not? Why? After all, I expect anyone who upgrades to
> 3.0
> > to run all of his unit tests to assert that everything still works (I
> expect
> > that to happen for every release, but for 3.0 in particular). Obviously
> the
> > runtime behaviors that were changed and documented in 2.9 are ones that
> he
> > might have already taken care of, but why can't he do the same reading
> the
> > CHANGES of 3.0?
> > I just feel that this policy forces us to think really hard and foresee
> > those changes in runtime behavior that we'd like to do in 3.0 so that we
> can
> > get them into 2.9, but at the end of the day we're not improving
> anything.
> > Upon upgrading to 2.9 I cannot handle the changes in runtime behaviors as
> > they weren't done yet. I can only do it after I upgrade to 3.0. So what's
> > the difference for me between fixing those that were documented in 2.9
> and
> > the new ones that were just released?
> >
> > Going forward, I don't think this community changes runtime behavior
> every
> > other Monday, and so I'd like to have the ability to make those changes
> > without such a strict policy. Those changes are meant to help our users
> (and
> > we are amongst them) to achieve better performance, usually, and so why
> > should we fear from making them, or if fear is too strong a word - why
> > should we refrain from doing them, while documenting the changes? If we
> > don't want to do it for every 'dot' release, we can do them in major
> > releases and I'd also vote for doing them in a mid-major release, like
> 3.5.
> >
> > Refactoring
> > ----------------
> > Today we are quite limited with refactoring. We cannot add methods to
> > interfaces or abstract methods to abstract classes, or even make classes
> > abstract. I'm perfectly fine with that as I don't want to face the need
> to
> > suddenly refactor my application just because Lucene decided to add a
> method
> > to an interface.
> >
> > But recently, while working on LUCENE-1593, me and Mike spotted a need to
> > add some methods to Weight, but since it is an interface we can't. So I
> said
> > something "let's do it in 3.0" but then we were not sure if this can be
> done
> > in 3.0. So the alternative was "let's deprecate Weight, create an
> > AbstractWeight class and do it there", but we weren't sure if that's even
> > something we can push for in 3.0, unless we do all of it in 2.9. This
> also
> > messes up the code, introducing new classes with bad names
> (AbstractWeight,
> > AbstractSearchable) where we could have avoided it if we just changed
> Weight
> > to an abstract class in 3.0.
> >
> > ---------------------------
> >
> > I think it all boils down to whether we MUST support jar drop-in-ability
> > when upgrading from 2.9 to 3.0. I think that we shouldn't as the whole
> > notion of 3.0 (or any future major version) is major revision to code,
> index
> > structure, JDK etc. If we're always expected to support it, then 2.9
> really
> > becomes 3.0 in terms of our ability to make changes to the API between
> > 2.9-3.0. I'm afraid that if that's the case, we might choose to hold on
> with
> > 2.9 as much as we can so we can push as many changes as we foresee into
> it,
> > so that they can be finalized in 3.0. I'm not sure what will that give
> us,
> > because upong upgrading to 2.9 I'll have to make a major revision to my
> code
> > if I want to simply drop the 3.0 jar in.
> >
> > And while back-compat of index structure is a MUST (i.e., I don't want to
> > rebuild my indexes every time I upgrade a major version), doing some
> house
> > cleaning to my code and taking advantage of the new classes and/or more
> > performing API is something I'm willing to live with.
> >
> > If the answer to all my questions is "we MUST do it this way because
> that's
> > what we've decided long ago", then so be it. But if we're willing to
> > re-consider our back-compat policy and maybe even change it, then it's
> > something we can put in the CHANGES of 2.9, going forward to 3.0 :)
> >
> > Shai
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Re: What are we allowed to do in 3.0?

[Michael McCandless <lu...@mikemccandless.com>]

I think 3.0 should be a fast turnaround after 2.9?  Ie, no "new
development" should take place?  We should remove deprecated APIs,
change defaults, etc., but that's about it.  (I think this is how past
major releases worked?).  It's a "fast switch".  Which then means we
need to do all the hard work in 2.9...

So, I think any API changes we want to make must be present in 2.9 as
deprecations.  We shouldn't up and remove / rename something in 3.0
with no fore-warning in 2.9.  Is there a case where this is too
painful?

Likewise I think we should give notification of expected changes in
runtime behavior with 2.9 (and not suddenly do them in 3.0).

> Which means upon releasing 2.9, the very first issue that's opened and committed should be removing the current deprecated methods, or otherwise we could open an issue that deprecates a method and accidentally remove it later, when we handle the massive deprecation removal.

I think we should not target JAR drop-inability, and we should allow
changes to runtime behavior, as well as certain minor API changes in
3.0.  EG here are some of the changes already slated for 3.0:

  * IndexReader.open returns readOnly reader by default

  * IndexReader.norms returns null on fields that don't have norms

  * InterruptedException is thrown by many APIs

  * IndexWriter.autoCommit is hardwired to false

  * Things that now return the deprecated IndexCommitPoint (interface)
    will be changed to return IndexCommit (abstract base class)

  * Directory.list will be removed; Directory.listAll will become an
    abstract method

  * Stop tracking scores by default when sorting by field

> But recently, while working on LUCENE-1593, me and Mike spotted a need to add some methods to Weight, but since it is an interface we can't. So I said something "let's do it in 3.0" but then we were not sure if this can be done in 3.0.

I think for this we should probably introduce an abstract base class
(implementing Weight) in 2.9, stating that Weight interface will be
removed in 3.0?  (EG, this is what was done for
IndexCommitPoint/IndexCommit).  Simply changing Weight to be an
abstract class in 3.0 is spooky because Java is single inheritance, ie
for existing classes that "implements Weight" but subclass something
else it would be nicer to give a heads up with 2.9 that they'll need
to refactor?

Mike

On Thu, Apr 30, 2009 at 6:25 AM, Shai Erera <se...@gmail.com> wrote:
> Hi
>
> Recently I was involved in several issues that required some runtime changes
> to be done in 3.0 and it was not so clear what is it that we're actually
> allowed to do in 3.0. So I'd like to open it for discussion, unless
> everybody agree on it already.
>
> So far I can tell that 3.0 allows us to:
> 1) Get rid of all the deprecated methods
> 2) Move to Java 1.5
>
> But what about changes to runtime behavior, re-factoring a whole set of
> classes etc? I'd like to relate to them one by one, so that we can comment
> on each.
>
> Removing deprecated methods
> ---------------------------------------------
> As I was told, 2.9 is the last release we are allowed to mark methods as
> deprecated, and remove them in 3.0. I.e., after 2.9 is out, if we feel there
> is a method that should be renamed, its signature should change or be
> removed altogether, we can't just do it and we'd have to deprecate it and
> remove it in 4.0 (?). I personally thought that 2.9 allows us to make these
> changes without letting anyone know about them in advance, which I'm ok with
> since upgrading to 3.0 is not going to be as 'smooth', but I also understand
> why 'letting people know in advance' (which means a release prior to the one
> when they are removed) gives a "better service" to our users. I also thought
> that jar drop-in-ability is not supposed to be supported from 2.9 to 3.0
> (but I was corrected previously on that).
> Which means upon releasing 2.9, the very first issue that's opened and
> committed should be removing the current deprecated methods, or otherwise we
> could open an issue that deprecates a method and accidentally remove it
> later, when we handle the massive deprecation removal. We should also create
> a 2.9 tag.
>
> Changes to runtime behavior
> -----------------------------------------
> What is exactly the policy here? If we document in 2.9 that certain
> features' runtime behavior will change in 3.0 - is that ok to make those
> changes? And if we don't document them and do it (in the transition from
> 2.9-3.0) then it's not? Why? After all, I expect anyone who upgrades to 3.0
> to run all of his unit tests to assert that everything still works (I expect
> that to happen for every release, but for 3.0 in particular). Obviously the
> runtime behaviors that were changed and documented in 2.9 are ones that he
> might have already taken care of, but why can't he do the same reading the
> CHANGES of 3.0?
> I just feel that this policy forces us to think really hard and foresee
> those changes in runtime behavior that we'd like to do in 3.0 so that we can
> get them into 2.9, but at the end of the day we're not improving anything.
> Upon upgrading to 2.9 I cannot handle the changes in runtime behaviors as
> they weren't done yet. I can only do it after I upgrade to 3.0. So what's
> the difference for me between fixing those that were documented in 2.9 and
> the new ones that were just released?
>
> Going forward, I don't think this community changes runtime behavior every
> other Monday, and so I'd like to have the ability to make those changes
> without such a strict policy. Those changes are meant to help our users (and
> we are amongst them) to achieve better performance, usually, and so why
> should we fear from making them, or if fear is too strong a word - why
> should we refrain from doing them, while documenting the changes? If we
> don't want to do it for every 'dot' release, we can do them in major
> releases and I'd also vote for doing them in a mid-major release, like 3.5.
>
> Refactoring
> ----------------
> Today we are quite limited with refactoring. We cannot add methods to
> interfaces or abstract methods to abstract classes, or even make classes
> abstract. I'm perfectly fine with that as I don't want to face the need to
> suddenly refactor my application just because Lucene decided to add a method
> to an interface.
>
> But recently, while working on LUCENE-1593, me and Mike spotted a need to
> add some methods to Weight, but since it is an interface we can't. So I said
> something "let's do it in 3.0" but then we were not sure if this can be done
> in 3.0. So the alternative was "let's deprecate Weight, create an
> AbstractWeight class and do it there", but we weren't sure if that's even
> something we can push for in 3.0, unless we do all of it in 2.9. This also
> messes up the code, introducing new classes with bad names (AbstractWeight,
> AbstractSearchable) where we could have avoided it if we just changed Weight
> to an abstract class in 3.0.
>
> ---------------------------
>
> I think it all boils down to whether we MUST support jar drop-in-ability
> when upgrading from 2.9 to 3.0. I think that we shouldn't as the whole
> notion of 3.0 (or any future major version) is major revision to code, index
> structure, JDK etc. If we're always expected to support it, then 2.9 really
> becomes 3.0 in terms of our ability to make changes to the API between
> 2.9-3.0. I'm afraid that if that's the case, we might choose to hold on with
> 2.9 as much as we can so we can push as many changes as we foresee into it,
> so that they can be finalized in 3.0. I'm not sure what will that give us,
> because upong upgrading to 2.9 I'll have to make a major revision to my code
> if I want to simply drop the 3.0 jar in.
>
> And while back-compat of index structure is a MUST (i.e., I don't want to
> rebuild my indexes every time I upgrade a major version), doing some house
> cleaning to my code and taking advantage of the new classes and/or more
> performing API is something I'm willing to live with.
>
> If the answer to all my questions is "we MUST do it this way because that's
> what we've decided long ago", then so be it. But if we're willing to
> re-consider our back-compat policy and maybe even change it, then it's
> something we can put in the CHANGES of 2.9, going forward to 3.0 :)
>
> Shai
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org