You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Simon Willnauer (JIRA)" <ji...@apache.org> on 2010/11/15 21:12:13 UTC
[jira] Created: (LUCENE-2764) Allow tests to use random codec per
field
Allow tests to use random codec per field
-----------------------------------------
Key: LUCENE-2764
URL: https://issues.apache.org/jira/browse/LUCENE-2764
Project: Lucene - Java
Issue Type: Test
Components: Tests
Affects Versions: 4.0
Reporter: Simon Willnauer
Priority: Minor
Fix For: 4.0
Attachments: LUCENE-2764.patch
Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-2764:
------------------------------------
Attachment: LUCENE-2764.patch
here is a first patch - all tests pass with it enabled
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934151#action_12934151 ]
Simon Willnauer commented on LUCENE-2764:
-----------------------------------------
bq. Nice find Mike!
oh man good catch :)
thanks for diggin
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764_alt.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-2764:
------------------------------------
Attachment: LUCENE-2764.patch
another iteration including suggestions from robert (thanks now that I get your point it makes lots of sense :D) to either run Random-PerField (without PreFlexRW) OR an PreFlex-Only codec provider randomly.
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933480#action_12933480 ]
Michael McCandless commented on LUCENE-2764:
--------------------------------------------
Patch looks great Simon! Since preflexrw is now "all or none" we don't need any changes to the impersonation logic...
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-2764:
------------------------------------
Attachment: LUCENE-2764.patch
Here is a new patch incorporating Roberts ideas. This patch enables the randomPerFieldCodec by default unless there is a explicit test codec set.
Eventually we would want to enable users to plug in their codecs into the RandomCodecProvider to test it against all our testcases. I think that should be a separate issue though.
simon
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933985#action_12933985 ]
Robert Muir commented on LUCENE-2764:
-------------------------------------
Nice find Mike!
I think this was a major cause of the problems!
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764_alt.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932593#action_12932593 ]
Simon Willnauer commented on LUCENE-2764:
-----------------------------------------
bq. anyway I think this is ready to be committed though - I will hold off a little and commit if nobody objects...
actually this doesn't use PreFlexRW since some tests don't like it - yet I think we need some way to exclude a codec within a testcase so that testcases that don't work with say PreFlex that this codec is not used within that testcase.
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932559#action_12932559 ]
Robert Muir commented on LUCENE-2764:
-------------------------------------
bq. Eventually we would want to enable users to plug in their codecs into the RandomCodecProvider to test it against all our testcases. I think that should be a separate issue though.
I don't think we should do this.
Our src/test shouldn't be a setup for people to test arbitrary extensions to lucene such as Directories or Codecs.
If it happens to work that they can simply edit a file and test these things, thats great, but we shouldnt officially support this in src/test.
src/test is no-holds-barred just doing the best we can to test lucene itself... it should stick to this level of scope.
if we want to provide mechanisms to test arbitrary user extensions it should be supported and documented in src/java like LockStressTest.
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933567#action_12933567 ]
Robert Muir commented on LUCENE-2764:
-------------------------------------
ok after working on this (and running the tests over and over again), i just hit a non-reproducible test fail.
this stuff is really complex, we should really consider whether we should do this.
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764_alt.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-2764:
------------------------------------
Attachment: LUCENE-2764.patch
s/TEST_RAMDOM_CODEC_PER_FIELD/TEST_RANDOM_CODEC_PER_FIELD
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932223#action_12932223 ]
Michael McCandless commented on LUCENE-2764:
--------------------------------------------
This is a great idea! Makes our tests even more random... though we may hit file descriptor limits in some tests...
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932600#action_12932600 ]
Simon Willnauer commented on LUCENE-2764:
-----------------------------------------
Well, I remember now why I didn't use PreFlexRW for perFieldRandom testing. The problem is if we have IW writing field A with codec Standard then open a new IW with field A using PreFlexRW we get problems with the comparator if those segments are merged though. So maybe we should make the whole thing disabled by default. and only run it optionally.
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933754#action_12933754 ]
Simon Willnauer commented on LUCENE-2764:
-----------------------------------------
bq. ok after working on this (and running the tests over and over again), i just hit a non-reproducible test fail.
robert I think your patch looks good though...
what kind of error did you hit? can you share?
simon
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764_alt.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-2764:
------------------------------------
Attachment: LUCENE-2764.patch
Here is a new patch that uses PreFlexRW in the RandomCodecProvider and all tests pass. I had to tell the backwards tests to use the default CodecProvider since otherwise the test will fail if some PreFlex PreFlexRW segments are merged in weird order.
feedback welcome
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2764:
--------------------------------
Attachment: LUCENE-2764_alt.patch
ok I think i reworked most of the problems: impersonation problems, etc.
patch is a bit messy probably but i think the logic is correct.
you can give it a try with 'ant test' (it changes a default of -Dtests.codec=randomPerField)
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764_alt.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-2764.
----------------------------------------
Resolution: Fixed
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764_alt.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932177#action_12932177 ]
Robert Muir commented on LUCENE-2764:
-------------------------------------
Simon, just an idea:
can we give the randomprovider a special -D value, such as 'randomPerField', and enable this by default?
We should set the value to 'randomPerField' both in lucenetestcase (for IDEs) and in common-build.xml/etc (for ANT) to be consistent.
additionally, is it possible to give this thing a toString() or similar, to dump out the field->codec mapping?
This would be useful when tests fail: the 'global' codec is currently printed in the @afterClass of LuceneTestCase:
{noformat}
// if tests failed, report some information back
if (testsFailed)
System.out.println("NOTE: test params are: codec=" + codec +
", locale=" + locale +
", timezone=" + (timeZone == null ? "(null)" : timeZone.getID()));
{noformat}
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer resolved LUCENE-2764.
-------------------------------------
Resolution: Fixed
Committed revision 1036543
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932591#action_12932591 ]
Simon Willnauer commented on LUCENE-2764:
-----------------------------------------
bq. if we want to provide mechanisms to test arbitrary user extensions it should be supported and documented in src/java like LockStressTest.
good point I guess you are right.
anyway I think this is ready to be committed though - I will hold off a little and commit if nobody objects...
simon
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2764:
---------------------------------------
Attachment: LUCENE-2764.patch
New version of this patch... there was a sneaky problem because IndexWriterConfig had a final static CodecProvider default, which held onto the wrong CodecProvider instance and caused tricky test failures.
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764_alt.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933442#action_12933442 ]
Robert Muir commented on LUCENE-2764:
-------------------------------------
Simon looks good.
this way, we randomly emulate working with lucene 3.0 indexes in our test, which are always completely preflex codec.
the patch looks good to me (i would fix the RAMDOM spelling to RANDOM in that one place, thats all).
It would probably be good to get Mike to check that "impersonation" is working [this is the whole business of how we use preflex-rw for writing but preflex for reading and they share the same name]. I think at one time, maybe for about 5 minutes i understood how this worked before, but i lost it.
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932910#action_12932910 ]
Michael McCandless commented on LUCENE-2764:
--------------------------------------------
bq. The problem is if we have IW writing field A with codec Standard then open a new IW with field A using PreFlexRW we get problems with the comparator if those segments are merged though.
Hmm this should be OK...
The PreFlexRW codec has a sneaky "impersonation" layer (test only)
that attempts to figure out which term comparator it's supposed to be
using when something is reading the segment. It sounds like that
layer isn't being smart enough now.
I think we could fix it -- really it just needs to know which codec is
"writing". If it's PreFlexRW that's writing then it needs to use the
legacy sort order; else, unicode.
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Reopened: (LUCENE-2764) Allow tests to use random codec per
field
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir reopened LUCENE-2764:
---------------------------------
I'm reopening as there are some problems... ill back out simon's patch and try to suggest some modifications
to fix the build but also improve reproducibility, etc.
> Allow tests to use random codec per field
> -----------------------------------------
>
> Key: LUCENE-2764
> URL: https://issues.apache.org/jira/browse/LUCENE-2764
> Project: Lucene - Java
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch, LUCENE-2764.patch
>
>
> Since we now have a real per field codec support we should enable to run the tests with a random codec per field. When I change something related to codecs internally I would like to ensure that whatever combination of codecs (except of preflex) I use the code works just fine. I created a RandomCodecProvider in LuceneTestCase that randomly selects the codec for fields when it sees them the first time. I disabled the test by default to leave the old randomize codec support in as it was / is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org