You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2010/01/15 00:21:54 UTC
[jira] Created: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
-------------------------------------------------------------------------------------------------------------------------------------------------
Key: LUCENE-2211
URL: https://issues.apache.org/jira/browse/LUCENE-2211
Project: Lucene - Java
Issue Type: Bug
Components: Analysis, contrib/analyzers
Affects Versions: 3.0
Reporter: Uwe Schindler
Fix For: 3.1
Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2211:
----------------------------------
Attachment: LUCENE-2211-branch30.patch
Patch for 3.0 branch
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211-branch30.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-2211) Improve BaseTokenStreamTestCase to
uses a fake attribute to check if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler reassigned LUCENE-2211:
-------------------------------------
Assignee: Uwe Schindler
> Improve BaseTokenStreamTestCase to uses a fake attribute to check if clearAttributes() was called correctly - found bugs in contrib/analyzers
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211-branch29.patch, LUCENE-2211-branch30.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2211:
----------------------------------
Attachment: LUCENE-2211-branch29.patch
Patch for 2.9 branch. Tests are running...
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211-branch29.patch, LUCENE-2211-branch30.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2211:
--------------------------------
Attachment: LUCENE-2211.patch
uwe's patch, with the fixes for contrib.
broken were compounds, n-gram filter, and edge n-gram filter
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2211:
----------------------------------
Attachment: LUCENE-2211.patch
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 3.0
> Reporter: Uwe Schindler
> Fix For: 3.1
>
> Attachments: LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2211:
--------------------------------
Attachment: LUCENE-2211.patch
Hi Uwe, PrefixAwareTokenFilter did not clearAttributes() either. I tried all others i could find but I think the rest are ok.
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Improve BaseTokenStreamTestCase to
uses a fake attribute to check if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2211:
----------------------------------
Summary: Improve BaseTokenStreamTestCase to uses a fake attribute to check if clearAttributes() was called correctly - found bugs in contrib/analyzers (was: Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers)
> Improve BaseTokenStreamTestCase to uses a fake attribute to check if clearAttributes() was called correctly - found bugs in contrib/analyzers
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211-branch29.patch, LUCENE-2211-branch30.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2211:
----------------------------------
Attachment: LUCENE-2211.patch
I'll commit attached patch woith some fixes for typos etc.
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2211) Advances BaseTokenStreamTestCase
that uses a fake attribute to check, if clearAttributes() was called
correctly - found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800698#action_12800698 ]
Uwe Schindler commented on LUCENE-2211:
---------------------------------------
Fixed in trunk revision: 899627
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2211:
----------------------------------
Affects Version/s: 2.9
2.9.1
Fix Version/s: 3.0.1
2.9.2
The ngram things are serious, so also backport.
We get the non-generic java 1.4 version for solr 1.5 for free.
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2211:
--------------------------------
Attachment: LUCENE-2211.patch
Hello Uwe, i did not get time to review all tokenstreams but I converted a ShingleMatrix test to assertTokenStreamContents and found a clearAttributes() bug in it too, so it is also fixed in this one, more tokenstreams with problems might remain.
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2211) Advances BaseTokenStreamTestCase
that uses a fake attribute to check, if clearAttributes() was called
correctly - found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800709#action_12800709 ]
Uwe Schindler commented on LUCENE-2211:
---------------------------------------
Fixed in Lucene 3.0 revision: 899639
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211-branch30.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2211) Advances BaseTokenStreamTestCase
that uses a fake attribute to check, if clearAttributes() was called
correctly - found bugs in contrib/analyzers
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800464#action_12800464 ]
Robert Muir commented on LUCENE-2211:
-------------------------------------
before committing any fix i want to review / add tests for any tokenstreams that do not yet use this BaseTokenStreamTestCase, just to be sure there are no others with this problem.
it may seem trivial but if this clearing does not take place properly, then things like position increment with stopfilter can grow to very large values, overflow, and cause IndexWriter to throw an exception: http://www.lucidimagination.com/search/document/f649a19901d33c75/illegalargumentexception_when_indexwriter_adddocument
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2211:
----------------------------------
Attachment: LUCENE-2211.patch
Some updates to TeeSink test. Changes.txt.
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2211) Improve BaseTokenStreamTestCase to
uses a fake attribute to check if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler resolved LUCENE-2211.
-----------------------------------
Resolution: Fixed
Committed into 2.9 branch revision: 899681
> Improve BaseTokenStreamTestCase to uses a fake attribute to check if clearAttributes() was called correctly - found bugs in contrib/analyzers
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211-branch29.patch, LUCENE-2211-branch29.patch, LUCENE-2211-branch30.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2211:
--------------------------------
Attachment: LUCENE-2211.patch
i reviewed all code with incrementToken() and found 3 more problems, 2 in highlighter, 1 in memory.
i also fixed all helper tokenstreams in core/contrib tests.
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Improve BaseTokenStreamTestCase to
uses a fake attribute to check if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2211:
----------------------------------
Attachment: LUCENE-2211-branch29.patch
There was a new bug in the 2.9 ShingleMatrixFilter because of incorrect Token reuse in a call to this.next(reusableToken). This patch now also contains the merged LUCENE-1939, which was missing.
I commit this now.
> Improve BaseTokenStreamTestCase to uses a fake attribute to check if clearAttributes() was called correctly - found bugs in contrib/analyzers
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211-branch29.patch, LUCENE-2211-branch29.patch, LUCENE-2211-branch30.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2211) Advances BaseTokenStreamTestCase that
uses a fake attribute to check, if clearAttributes() was called correctly -
found bugs in contrib/analyzers
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-2211:
----------------------------------
Attachment: LUCENE-2211.patch
More updates to TeeSink and also BaseTokenStreamTestCase to still call clearAttributes to force TS to not reuse attributes from previous calls to incrementToken(). This was one of the first traps, Robert investigated in 2.9.
> Advances BaseTokenStreamTestCase that uses a fake attribute to check, if clearAttributes() was called correctly - found bugs in contrib/analyzers
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2211
> URL: https://issues.apache.org/jira/browse/LUCENE-2211
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis, contrib/analyzers
> Affects Versions: 2.9, 2.9.1, 3.0
> Reporter: Uwe Schindler
> Fix For: 2.9.2, 3.0.1, 3.1
>
> Attachments: LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch, LUCENE-2211.patch
>
>
> Robert had the idea to use a fake attribute inside BaseTokenStreamTestCase that records if its clear() method was called. If this is not the case after incrementToken(), asserTokenStreamContents fails. It also uses the attribute in TeeSinkTokenFilter, because there a lot of copying, captureState and restoreState() is used. By the attribute, you can track wonderful, if save/restore and clearAttributes is correctly implemented. It also verifies that *before* a captureState() it was also cleared (as the state will also contain the clear call). Because if you consume tokens in a filter, capture the consumed tokens and insert them, the capturedStates must also be cleared before.
> In contrib analyzers are some test that fail to pass this additional assertion. They are not fixed in the attached patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org