You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Wouter Heijke (JIRA)" <ji...@apache.org> on 2009/05/14 09:52:45 UTC
[jira] Created: (LUCENE-1636) TokenFilters with a null value in the
constructor fail
TokenFilters with a null value in the constructor fail
------------------------------------------------------
Key: LUCENE-1636
URL: https://issues.apache.org/jira/browse/LUCENE-1636
Project: Lucene - Java
Issue Type: Bug
Affects Versions: 2.9
Reporter: Wouter Heijke
While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
One problem is with TokenFilters that do a super(null) in the constructor.
I fixed it by changing the constructor to super(new EmptyTokenStream())
This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1636) TokenFilters with a null value in the
constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-1636:
----------------------------------
Attachment: LUCENE-1636.patch
This patch prevents changing the delegate stream.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Wouter Heijke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712660#action_12712660 ]
Wouter Heijke commented on LUCENE-1636:
---------------------------------------
A good example of this 'abuse' is in ShingleMatrixFilter.java in one of it's constructors.
Here also the input is not used and replaced by an EmptyTokenStream:
// set the input to be an empty token stream, we already have the data.
this.input = new EmptyTokenStream();
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712025#action_12712025 ]
Uwe Schindler commented on LUCENE-1636:
---------------------------------------
Oh, you already committed this :)
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Wouter Heijke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711915#action_12711915 ]
Wouter Heijke commented on LUCENE-1636:
---------------------------------------
I only hope users will understand this and they realize that 2.9 is not backwards compatible to previous versions! This code in our codebase was added by knowledgeable Lucene developers!
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1636.
----------------------------------------
Resolution: Fixed
Fix Version/s: 2.9
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Wouter Heijke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711933#action_12711933 ]
Wouter Heijke commented on LUCENE-1636:
---------------------------------------
I'm on holiday now, but as far as I recollect (as I was not the author of the code) it was done on some filters that would be used in another situation (similar to a filter) to use the filter's functionality. Also it was used with filters that could not be extended, so a new filter was created, also here the orignal filter's public methods would be called.
In a way it doesn't matter, it could be done with the api without any problems with the latest few releases that i know of.
A more elegant way if one would like to introduce this new behaviour is to at least log some kind of error message in the 2.9 release so users would be alarmed that they use the Lucene api in a way that is not supported anymore.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711704#action_12711704 ]
Michael McCandless commented on LUCENE-1636:
--------------------------------------------
I think we should change this in 2.9, for the reasons Uwe pointed out, to disallow changing the delegate after construction.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709375#action_12709375 ]
Uwe Schindler commented on LUCENE-1636:
---------------------------------------
What do you want to do with an filter without a delegate? Calling next() would simply NPE, too!
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Reopened: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reopened LUCENE-1636:
----------------------------------------
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762387#action_12762387 ]
Uwe Schindler commented on LUCENE-1636:
---------------------------------------
This change is also noted in the backwards compatibility section of Lucene 2.9.
The assignment of filter in the ctor is totally useless, as the super ctor does it already, so it the problem of this third party software that used the API in an undocumented way. I am sorry for your problems, but the author of lucene-ja should provide a fix, if you have the source code available, it is a less important problem, if it is closed source, you have to ask the author to fix it soon.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Mark Bennett (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762281#action_12762281 ]
Mark Bennett commented on LUCENE-1636:
--------------------------------------
(trouble posting, forgive if duplicate)
This change also broke the Japanese morphological SEN / Lucene integration code in lucene-ja. Since Solr 1.4 is based on Lucene 2.9, this will also effectively break SEN for Solr users who upgrade to 1.4.
I'm not complaining. Reading the above comments, the change was probably the "right" thing to do. I've contacted the author of lucene-ja, and I hope to work on a rewrite to address this.
I would be interested in any comments you folks might have about the lucene-ja code.
Class org.apache.lucene.analysis.ja.POSFilter
Extends org.apache.lucene.analysis.TokenFilter
Offending code in lucene-ja's POSFilter
public POSFilter(TokenStream in, Hashtable posTable) {
super(in);
input = in; // <<==== this is a member field of parent TokenFilter
table = posTable;
}
This is done in several classes.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Mark Bennett (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763303#action_12763303 ]
Mark Bennett commented on LUCENE-1636:
--------------------------------------
Uwe,
Your comments were a home run, thanks!
There are 6 places in the code where constructors needlessly try to assign that. Commenting them all out fixes that problem.
There's some other stuff to clean up and I'm chatting with the author.
Thanks again,
Mark
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712671#action_12712671 ]
Uwe Schindler commented on LUCENE-1636:
---------------------------------------
bq. A good example of this 'abuse' is in ShingleMatrixFilter.java in one of it's constructors.
This class extends TokenStream not TokenFilter! As the input instance member of TokenFilter is now final, it would not even compile. Please note, this is one of the contrib packages, not yet using the new API, so with useNewApi set to true, this TokenStream would fail (see LUCENE-1460). The change, you have the problem with, is caused by the new TokenStream API, and so NULL delegates are not possible!
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Wouter Heijke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709371#action_12709371 ]
Wouter Heijke commented on LUCENE-1636:
---------------------------------------
Here's an example
before:
private MyFilter myFilter = new MyFilter(null);
after:
private MyFilter myFilter = new MyFilter(new EmptyTokenStream());
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Wouter Heijke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709353#action_12709353 ]
Wouter Heijke commented on LUCENE-1636:
---------------------------------------
My own filters and my own tests.
It is just that the behaviour of Lucene has changed in such cases.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712076#action_12712076 ]
Michael McCandless commented on LUCENE-1636:
--------------------------------------------
OK I'll add a null check in AttributeSource's ctor.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1636.
----------------------------------------
Resolution: Won't Fix
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-1636) TokenFilters with a null
value in the constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711920#action_12711920 ]
Uwe Schindler edited comment on LUCENE-1636 at 5/21/09 6:53 PM:
----------------------------------------------------------------
Mike:
Would this affect backwards compatibility? If we make it final and nobody changes the stream, everything is ok. Is this also the case, when using lucene.jar as a dropin-replacement without recompilation? Will changing a final variable from code, compiled before finalization, be detected by the JVM? Is the compiled code with final still binary compatible to code compiled againt non-final members?
I think, we should try this out before committing!
was (Author: thetaphi):
Mike:
Would this affect backwards compatibility? If we make it final and nobody changes the stream, everything is ok. Is this also the case, when using plugin-replacement without recompilation? Will changing a final variable from code, compiled before finalization, be detected by the JVM? Is the compiled code with final still binary compatible to code compiled againt non-final members?
I think, we should try this out before committing!
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711920#action_12711920 ]
Uwe Schindler commented on LUCENE-1636:
---------------------------------------
Mike:
Would this affect backwards compatibility? If we make it final and nobody changes the stream, everything is ok. Is this also the case, when using plugin-replacement without recompilation? Will changing a final variable from code, compiled before finalization, be detected by the JVM? Is the compiled code with final still binary compatible to code compiled againt non-final members?
I think, we should try this out before committing!
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Wouter Heijke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709372#action_12709372 ]
Wouter Heijke commented on LUCENE-1636:
---------------------------------------
Whatever cause it was, it was behaving this way for a long time, changing it to the new way in 3.0 seems more acceptable to me then in 2.9 to me.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712021#action_12712021 ]
Uwe Schindler commented on LUCENE-1636:
---------------------------------------
Thanks, I am still in Japan and had no time to check this out myself. So it seems good. I would suggest to add this to changes.txt with a warning about the change in backwards compatibility.
The issue itsself should be closed as wont-fix, as the usage of Filters with a null delegate is not specified in API and is not correct.
The case, Wouter explained, is just a workaround for incorrect usage of filters. He wants to reuse some methods of the filter somewhere else and wants to construct a "dummy" instance for that. He should not do this and better move the code accessible outside of the TokenFilter to a separate class.
Maybe we should add a special ==null test to the constructor, that early throws a NPE with a short notice, that this usage is no longer supported.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711919#action_12711919 ]
Uwe Schindler commented on LUCENE-1636:
---------------------------------------
Hi Wouter,
I still want to find out, what you are trying to do with a TokenFilter without a delegate! Can you explain, why you want to initialize with super(null)?
If it is because you want to change it later to something non-null, it will not work anymore (this is why I want to make the delgate stream final). So please explain!
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712008#action_12712008 ]
Michael McCandless commented on LUCENE-1636:
--------------------------------------------
Good questions Uwe!
I tested the back-compat by adding this to TestAnalyzers temporarily
in my local checkout:
{code}
private class TestFilter extends TokenFilter {
public TestFilter() {
super(new WhitespaceTokenizer(null));
}
}
public void testChangeTokenFilterInput() {
TokenFilter tf = new TestFilter();
System.out.println("tf.input = " + tf.input);
tf.input = null;
}
{code}
Then, {{ant test-tag -Dtestcase=TestAnalyzers}} results in:
{code}
[junit] ------------- Standard Output ---------------
[junit] tf.input = (start=0,end=0,term=)
[junit] ------------- ---------------- ---------------
[junit] Testcase: testChangeTokenFilterInput(org.apache.lucene.analysis.TestAnalyzers): Caused an ERROR
[junit] null
[junit] java.lang.IllegalAccessError
[junit] at org.apache.lucene.analysis.TestAnalyzers.testChangeTokenFilterInput(TestAnalyzers.java:143)
{code}
So, 1) it doesn't break JAR drop-in-abilility when one references
input, and 2) indeed at runtime final-ness is enforced by the JRE. So
I think we should proceed with the change? It is a back-compat break
for those users who change input after creating a TokenFilter, but
such a use case was not legal usage of the API, and will specifically
not work like it did in the past (so back-compat was already broken,
just in a much more sneaky manner).
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-1636:
------------------------------------------
Assignee: Michael McCandless
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Assignee: Michael McCandless
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709369#action_12709369 ]
Uwe Schindler commented on LUCENE-1636:
---------------------------------------
Hi Wouter,
I tend to say, this is Won't fix:
- You use TokenFilter in an undocumented way. I suspect, that you set the delegate stream later, correct? To prevent this, the protected "input" member should normally be "final", as this is not possible (but worked with old analyzers) - ok, this was an error, to not make the delegate strean final (but we shoudl fix this *now*)
- The main problem of changing the non-final delegate stream later lies in the fact, that with 2.9 a new TokenStream API (setUseNewAPI()) was added. This new API does not use the Token class anymore, but uses Attributes. The stream's attributes list must be reused in the TokenFilters. If you initialize a TokenFilter with a specific delegate, the attribute instances are copied. If you change the delegate stream later, the filter still uses the attributes of the original stream and will update the wrong ones, it will not work anymore.
So we cannot support this wrong behaviour of TokenFilter anymore. But we should fix the final accessor of "input", I will attach a patch.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709348#action_12709348 ]
Michael McCandless commented on LUCENE-1636:
--------------------------------------------
Hmm... that's no good.
Can you give some more specifics here? Which TokenFilter(s) are doing super(null)?
What other unit tests are failing?
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1636) TokenFilters with a null value in
the constructor fail
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709374#action_12709374 ]
Uwe Schindler commented on LUCENE-1636:
---------------------------------------
If you do not use a delegate TokenStream, extend TokenStream instead of TokenFilter. If you want to later set the delegate, see my comment (and patch to prevent this incorrect usage) above.
> TokenFilters with a null value in the constructor fail
> ------------------------------------------------------
>
> Key: LUCENE-1636
> URL: https://issues.apache.org/jira/browse/LUCENE-1636
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Wouter Heijke
> Attachments: LUCENE-1636.patch
>
>
> While migrating from 2.4.x to 2.9-dev I found a lot of failing unittests.
> One problem is with TokenFilters that do a super(null) in the constructor.
> I fixed it by changing the constructor to super(new EmptyTokenStream())
> This will cause problems and frustration to others while migrating to 2.9.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org