You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2010/07/22 21:24:49 UTC
[jira] Created: (LUCENE-2554) preflex codec doesn't order terms
correctly
preflex codec doesn't order terms correctly
-------------------------------------------
Key: LUCENE-2554
URL: https://issues.apache.org/jira/browse/LUCENE-2554
Project: Lucene - Java
Issue Type: Test
Reporter: Michael McCandless
Assignee: Michael McCandless
Fix For: 4.0
The surrogate dance in the preflex codec (which must dynamically remap terms from UTF16 order to unicode code point order) is buggy.
To better test it, I want to add a test-only codec, preflexrw, that is able to write indices in the pre-flex format. Then we should also fix tests to randomly pick codecs (including preflexrw) so we better test all of our codecs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2554) preflex codec doesn't order terms
correctly
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2554:
--------------------------------
Attachment: LUCENE-2554_merge.patch
here is a patch of the merge to trunk.
All tests pass.
> preflex codec doesn't order terms correctly
> -------------------------------------------
>
> Key: LUCENE-2554
> URL: https://issues.apache.org/jira/browse/LUCENE-2554
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 4.0
>
> Attachments: LUCENE-2554.patch, LUCENE-2554.patch, LUCENE-2554.patch, LUCENE-2554_merge.patch
>
>
> The surrogate dance in the preflex codec (which must dynamically remap terms from UTF16 order to unicode code point order) is buggy.
> To better test it, I want to add a test-only codec, preflexrw, that is able to write indices in the pre-flex format. Then we should also fix tests to randomly pick codecs (including preflexrw) so we better test all of our codecs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2554) preflex codec doesn't order terms
correctly
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891585#action_12891585 ]
Robert Muir commented on LUCENE-2554:
-------------------------------------
bq. We still need a clean way to randomly swap in the preflexrw codec....
I don't think we should do this with RandomIndexWriter like we do now, but pull this stuff out of there and move it to ant/LuceneTestCase.
I would prefer if we could supply a variable to ant (e.g. -Dtest.codec=) and LuceneTestCase[J4] would set the codec to this.
We could allow for a value of "random" here also to do what RIW does today.
I think this would make it easier to run the entire test suite with different codecs.
Also, some tests testCases might not be suitable for all codecs, and so we need to add annotations or some way to special-case these tests.
> preflex codec doesn't order terms correctly
> -------------------------------------------
>
> Key: LUCENE-2554
> URL: https://issues.apache.org/jira/browse/LUCENE-2554
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 4.0
>
> Attachments: LUCENE-2554.patch, LUCENE-2554.patch, LUCENE-2554.patch
>
>
> The surrogate dance in the preflex codec (which must dynamically remap terms from UTF16 order to unicode code point order) is buggy.
> To better test it, I want to add a test-only codec, preflexrw, that is able to write indices in the pre-flex format. Then we should also fix tests to randomly pick codecs (including preflexrw) so we better test all of our codecs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2554) preflex codec doesn't order terms
correctly
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-2554.
---------------------------------
Resolution: Fixed
Committed revision 979453.
> preflex codec doesn't order terms correctly
> -------------------------------------------
>
> Key: LUCENE-2554
> URL: https://issues.apache.org/jira/browse/LUCENE-2554
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 4.0
>
> Attachments: LUCENE-2554.patch, LUCENE-2554.patch, LUCENE-2554.patch, LUCENE-2554_merge.patch
>
>
> The surrogate dance in the preflex codec (which must dynamically remap terms from UTF16 order to unicode code point order) is buggy.
> To better test it, I want to add a test-only codec, preflexrw, that is able to write indices in the pre-flex format. Then we should also fix tests to randomly pick codecs (including preflexrw) so we better test all of our codecs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2554) preflex codec doesn't order terms
correctly
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2554:
---------------------------------------
Attachment: LUCENE-2554.patch
Fixed a couple more issues found by intensive overnight random testing... cleaned up a few things.
We still need a clean way to randomly swap in the preflexrw codec....
> preflex codec doesn't order terms correctly
> -------------------------------------------
>
> Key: LUCENE-2554
> URL: https://issues.apache.org/jira/browse/LUCENE-2554
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 4.0
>
> Attachments: LUCENE-2554.patch, LUCENE-2554.patch, LUCENE-2554.patch
>
>
> The surrogate dance in the preflex codec (which must dynamically remap terms from UTF16 order to unicode code point order) is buggy.
> To better test it, I want to add a test-only codec, preflexrw, that is able to write indices in the pre-flex format. Then we should also fix tests to randomly pick codecs (including preflexrw) so we better test all of our codecs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2554) preflex codec doesn't order terms
correctly
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891364#action_12891364 ]
Robert Muir commented on LUCENE-2554:
-------------------------------------
the perf issues here are really from our contrived tests... its good to use _TestUtil.randomUnicodeString, but it gives you the impression there is something wrong with this dance and there really isnt.
I added _TestUtil.randomRealisticUnicodeString in r966878, you can swap this into some of these slow tests and see its definitely the problem.
> preflex codec doesn't order terms correctly
> -------------------------------------------
>
> Key: LUCENE-2554
> URL: https://issues.apache.org/jira/browse/LUCENE-2554
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 4.0
>
> Attachments: LUCENE-2554.patch
>
>
> The surrogate dance in the preflex codec (which must dynamically remap terms from UTF16 order to unicode code point order) is buggy.
> To better test it, I want to add a test-only codec, preflexrw, that is able to write indices in the pre-flex format. Then we should also fix tests to randomly pick codecs (including preflexrw) so we better test all of our codecs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2554) preflex codec doesn't order terms
correctly
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2554:
---------------------------------------
Attachment: LUCENE-2554.patch
Fixed the test failures -- all tests should pass.
> preflex codec doesn't order terms correctly
> -------------------------------------------
>
> Key: LUCENE-2554
> URL: https://issues.apache.org/jira/browse/LUCENE-2554
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 4.0
>
> Attachments: LUCENE-2554.patch, LUCENE-2554.patch
>
>
> The surrogate dance in the preflex codec (which must dynamically remap terms from UTF16 order to unicode code point order) is buggy.
> To better test it, I want to add a test-only codec, preflexrw, that is able to write indices in the pre-flex format. Then we should also fix tests to randomly pick codecs (including preflexrw) so we better test all of our codecs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2554) preflex codec doesn't order terms
correctly
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2554:
---------------------------------------
Attachment: LUCENE-2554.patch
Attached VERY rough patch.
Mosts tests pass but at least 2 don't.
I think the fixes to surrogates dance are close to correct. The new random test (TestSurrogates) is good at finding bugs there...
The hack I have to forcefully use the test-only preflexrw codec needs to be made more generic, so that we can randomly pick the default codec to use for each test.
> preflex codec doesn't order terms correctly
> -------------------------------------------
>
> Key: LUCENE-2554
> URL: https://issues.apache.org/jira/browse/LUCENE-2554
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 4.0
>
> Attachments: LUCENE-2554.patch
>
>
> The surrogate dance in the preflex codec (which must dynamically remap terms from UTF16 order to unicode code point order) is buggy.
> To better test it, I want to add a test-only codec, preflexrw, that is able to write indices in the pre-flex format. Then we should also fix tests to randomly pick codecs (including preflexrw) so we better test all of our codecs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org