You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Simon Willnauer (JIRA)" <ji...@apache.org> on 2012/05/29 21:37:23 UTC
[jira] [Created] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Simon Willnauer created LUCENE-4087:
---------------------------------------
Summary: Provide consistent IW behavior for illegal meta data changes
Key: LUCENE-4087
URL: https://issues.apache.org/jira/browse/LUCENE-4087
Project: Lucene - Java
Issue Type: Improvement
Components: core/index
Affects Versions: 4.0
Reporter: Simon Willnauer
Fix For: 4.0, 4.1, 5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291613#comment-13291613 ]
Robert Muir commented on LUCENE-4087:
-------------------------------------
I like this approach:
* I think we should not view this behavior as fixed in stone. If we decide in the future we should
be able to change this kind of stuff in even a minor release if we feel like it. DocValues is a
*typed* field (thats its whole point!) and you shouldnt be mixing up types.
* I think we should fix this in the way you propose so we can get a 4.0 alpha release out!
Some comments:
* I think we don't need anything like this in NormsConsumer, because it looks like we have a check
already for this... right? Of course its much trickier for someone to changeup types there.
* I think somewhere in the documentation we should have a line of javadoc "docvalues are a typed
field, if you mix up types the end result is undefined". This way we don't have to commit to
any specific behavior as mentioned above.
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: LUCENE-4087.patch
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291742#comment-13291742 ]
Michael McCandless commented on LUCENE-4087:
--------------------------------------------
I think this is a good baby step for 4.0.
But I think it's important the javadocs make it clear that if you change up the DV type for a given field, the behavior is undefined and we are free to improve it in the future.
Ideally I think apps should get clear exceptions on attempting to index a doc with an incompatible change to anything that is our effective schema (omitNorms, indexOptions, DV types, etc.). For example, if a given field already omitNorms and you try to add a doc with that field not omitting norms, you should get a clear exception (it can only be an app bug, because on merge the norms will silently go away). But let's open a separate issue for that...
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: LUCENE-4087.patch
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer resolved LUCENE-4087.
-------------------------------------
Resolution: Fixed
Assignee: Simon Willnauer
Lucene Fields: New,Patch Available (was: New)
committed to trunk and 4x
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: LUCENE-4087.patch, LUCENE-4087.patch
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4087:
------------------------------------
Attachment: LUCENE-4087.patch
here is a new patch adding documentation to DocValues.java & a reference to all DV Fields. I added a bunch of tests including verification test that changing norms types actually fails. I extended the type promoter a little to actually promote INT_16 & INT_8 to Float32 if needed as well as INT_32 to FLOAT_64.
I think its ready
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: LUCENE-4087.patch, LUCENE-4087.patch
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291940#comment-13291940 ]
Robert Muir commented on LUCENE-4087:
-------------------------------------
nit: loosing -> losing in DocValues.java javadocs
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: LUCENE-4087.patch, LUCENE-4087.patch
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291972#comment-13291972 ]
Michael McCandless commented on LUCENE-4087:
--------------------------------------------
OK I guess that makes sense. Basically we sign up, now, to allow certain DV type changes in the schema, just like how we allow omitNorms to change from false to true, but not vice/versa.
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: LUCENE-4087.patch, LUCENE-4087.patch
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4087:
------------------------------------
Attachment: LUCENE-4087.patch
This patch implements a solution for docvalues that tries best effort if and only if the given value can be indexed without loosing any information. The DV type of the indexed field might be different to the "stored" one but we won't loose any information. This patch also handles this per DWPT and not global. The idea is that type promotion will fix the diverging types during merges but we guarantee persistence per segment.
for example if somebody indexed VAR_INTS but adds a FIXED_INT_* afterwards it should work just fine. While if somebody starts with FIXED_INT_8 and moves up to FIXED_INT_16 IW throws a IllegalArgumentException.
The patch is still missing javadocs here and there but given the controversial nature of this issue I wanna put it up for discusssion.
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: LUCENE-4087.patch
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291946#comment-13291946 ]
Simon Willnauer commented on LUCENE-4087:
-----------------------------------------
{quote}
Can we shorten the javadoc to simply state "Changing the DocValue type
for a given field is not supported."? Sure we make best effort to
"recover" today but I don't think we should detail particulars of the
specific best effort we're doing in 4.0?
{quote}
I am not sure if we should say that since its not true. you can safely change a float into a double and if you reindex all documents you will eventually converge to double. Same is true for Sorted and go from fixed to variable or extend the precision of an integer. I think its just fair to document that. if we can change it in future releases is a different thing.
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: LUCENE-4087.patch, LUCENE-4087.patch
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291600#comment-13291600 ]
Simon Willnauer commented on LUCENE-4087:
-----------------------------------------
After talking about this to other committers during the conference I think this is really a bit more controversial than it seemed. Except of the DocValues behavior this all is pre-existing behavior. The discussion is similar to changing norms through IR and removing that capability did bring up some hard discussions. Yet, I think we should only solve the DocValues issue in the least intrusive way and discuss the omitNorms & IndexOptions behavior in a different issue. If we make all this throwing exceptions we almost introduce a schema here which makes lucene 4.0 very different in terms of RT behavior compared to previous versions.
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Simon Willnauer updated LUCENE-4087:
------------------------------------
Description:
Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
we can approach this similar to how we handle consistent field number and:
* throw exception if indexOptions conflict (e.g. omitTF=true versus
false) instead of silently dropping positions on merge
* same with omitNorms
* same with norms types and docvalues types
* still keeping field numbers consistent
this way we could eliminate all these traps and just give an
exception instead.
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4087) Provide consistent IW behavior for
illegal meta data changes
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291937#comment-13291937 ]
Michael McCandless commented on LUCENE-4087:
--------------------------------------------
Patch looks good. +1
I think this patch now means that, depending on when flushes kick in,
you can sometimes apparently succeed in changing DV type for a field
(though on merge something strange can happen, eg suddenly "upgrading"
to a BYTES_XXX type) and other times hit an exception? Like the error
checking is now "intermittent" as seen from the app? You might think
everything is OK, push to production, and later (in production) hit a
new exception...
I think that's actually OK for now (this is all
best effort)... but I think we should clean this up (can come after 4.0)
so that the checking is consistent.
Can we shorten the javadoc to simply state "Changing the DocValue type
for a given field is not supported."? Sure we make best effort to
"recover" today but I don't think we should detail particulars of the
specific best effort we're doing in 4.0?
> Provide consistent IW behavior for illegal meta data changes
> ------------------------------------------------------------
>
> Key: LUCENE-4087
> URL: https://issues.apache.org/jira/browse/LUCENE-4087
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Fix For: 4.0, 4.1, 5.0
>
> Attachments: LUCENE-4087.patch, LUCENE-4087.patch
>
>
> Currently IW fails late and inconsistent if field metadata like an already defined DocValues type or "un"-omitting norms.
> we can approach this similar to how we handle consistent field number and:
>
> * throw exception if indexOptions conflict (e.g. omitTF=true versus
> false) instead of silently dropping positions on merge
> * same with omitNorms
> * same with norms types and docvalues types
> * still keeping field numbers consistent
> this way we could eliminate all these traps and just give an
> exception instead.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org