You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2011/07/13 18:08:59 UTC
[jira] [Created] (LUCENE-3309) Add narrow API for loading stored
fields, to replace FieldSelector
Add narrow API for loading stored fields, to replace FieldSelector
------------------------------------------------------------------
Key: LUCENE-3309
URL: https://issues.apache.org/jira/browse/LUCENE-3309
Project: Lucene - Java
Issue Type: Improvement
Components: core/index
Reporter: Michael McCandless
Fix For: Field Type branch
I think we should "invert" the FieldSelector API, with a "push" API
whereby FieldsReader invokes this API once per field in the document
being visited.
Implementations of the API can then do arbitrary things like save away
the field's size, load the field, clone the IndexInput for later lazy
loading, etc.
This very thin API would be a mirror image of the very thin index time
API we now have (IndexableField) and, importantly, it would have no
dependence on our "user space" Document/Field/FieldType impl, so apps
are free to do something totally custom.
After we have this, we should build the "sugar" API that rebuilds a
Document instance (ie IR.document(int docID)) on top of this new thin
API. This'll also be a good test that the API is sufficient.
Relevant discussions from IRC this morning at
http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-07-13#l76
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3309) Add narrow API for loading stored
fields, to replace FieldSelector
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3309:
---------------------------------------
Attachment: LUCENE-3309.patch
New patch, this time moving *FieldSelector (including LazyField) to
contrib/misc, and adding a "bridge class" (FieldSelectorVisitor) that
implements the new FieldVisitor, wrapping a FieldSelector.
FieldsReader is now very simple -- the only API it supports for
reading a doc is visitDocument (taking FieldsVisitor). It now knows
nothing about Document/Field (same w/ the IR impls; only IR base class
knows about Document/Field since it impls the sugar API).
Tests pass; I think it's ready to commit!
> Add narrow API for loading stored fields, to replace FieldSelector
> ------------------------------------------------------------------
>
> Key: LUCENE-3309
> URL: https://issues.apache.org/jira/browse/LUCENE-3309
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Fix For: Field Type branch
>
> Attachments: LUCENE-3309.patch, LUCENE-3309.patch
>
>
> I think we should "invert" the FieldSelector API, with a "push" API
> whereby FieldsReader invokes this API once per field in the document
> being visited.
> Implementations of the API can then do arbitrary things like save away
> the field's size, load the field, clone the IndexInput for later lazy
> loading, etc.
> This very thin API would be a mirror image of the very thin index time
> API we now have (IndexableField) and, importantly, it would have no
> dependence on our "user space" Document/Field/FieldType impl, so apps
> are free to do something totally custom.
> After we have this, we should build the "sugar" API that rebuilds a
> Document instance (ie IR.document(int docID)) on top of this new thin
> API. This'll also be a good test that the API is sufficient.
> Relevant discussions from IRC this morning at
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-07-13#l76
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3309) Add narrow API for loading stored
fields, to replace FieldSelector
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065188#comment-13065188 ]
Simon Willnauer commented on LUCENE-3309:
-----------------------------------------
ah man I am to late :)
I only looked at the StoredFieldVisitor briefly, what I always missed is a String... fieldsToLoad ctor for convenience which simply passes an array that is used to create a set. in several environments this would be helpful for example when you get the fields to load from a request or something like that.
I also don't understand why the set is null in the default ctors, can't we use the empty set here. If we want to prevent the method call if no fields should be loaded there should be a LoadNoFieldsVisitor I think, no?
> Add narrow API for loading stored fields, to replace FieldSelector
> ------------------------------------------------------------------
>
> Key: LUCENE-3309
> URL: https://issues.apache.org/jira/browse/LUCENE-3309
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Fix For: Field Type branch
>
> Attachments: LUCENE-3309.patch, LUCENE-3309.patch, LUCENE-3309.patch
>
>
> I think we should "invert" the FieldSelector API, with a "push" API
> whereby FieldsReader invokes this API once per field in the document
> being visited.
> Implementations of the API can then do arbitrary things like save away
> the field's size, load the field, clone the IndexInput for later lazy
> loading, etc.
> This very thin API would be a mirror image of the very thin index time
> API we now have (IndexableField) and, importantly, it would have no
> dependence on our "user space" Document/Field/FieldType impl, so apps
> are free to do something totally custom.
> After we have this, we should build the "sugar" API that rebuilds a
> Document instance (ie IR.document(int docID)) on top of this new thin
> API. This'll also be a good test that the API is sufficient.
> Relevant discussions from IRC this morning at
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-07-13#l76
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3309) Add narrow API for loading stored
fields, to replace FieldSelector
Posted by "Chris Male (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065165#comment-13065165 ]
Chris Male commented on LUCENE-3309:
------------------------------------
Super, +1 to committing.
> Add narrow API for loading stored fields, to replace FieldSelector
> ------------------------------------------------------------------
>
> Key: LUCENE-3309
> URL: https://issues.apache.org/jira/browse/LUCENE-3309
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Fix For: Field Type branch
>
> Attachments: LUCENE-3309.patch, LUCENE-3309.patch, LUCENE-3309.patch
>
>
> I think we should "invert" the FieldSelector API, with a "push" API
> whereby FieldsReader invokes this API once per field in the document
> being visited.
> Implementations of the API can then do arbitrary things like save away
> the field's size, load the field, clone the IndexInput for later lazy
> loading, etc.
> This very thin API would be a mirror image of the very thin index time
> API we now have (IndexableField) and, importantly, it would have no
> dependence on our "user space" Document/Field/FieldType impl, so apps
> are free to do something totally custom.
> After we have this, we should build the "sugar" API that rebuilds a
> Document instance (ie IR.document(int docID)) on top of this new thin
> API. This'll also be a good test that the API is sufficient.
> Relevant discussions from IRC this morning at
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-07-13#l76
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3309) Add narrow API for loading stored
fields, to replace FieldSelector
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3309:
---------------------------------------
Attachment: LUCENE-3309.patch
Initial patch:
- Adds IndexReader.FieldVisitor and abstract
IndexReader.document(int docID, FieldVisitor visitor). IR
subclasses and FieldsReader implement this.
- Stop storing tokenized bit in stored fields -- the fast vector
highlighter was actually using this metadata; I change it to
always insert the separator char (even for not-analyzed fields)
and changed the keyword analyzer in the tests to set offset gap
to 1. We need to do this anyway for LUCENE-2309, and since we are
making an API here we may as well remove it now...
- Implement IndexReader.document(int) using only the new
FieldVisitor API.
All tests pass.
I think next up is to cutover the various FieldSelector impls to
FieldVisitor impls, move them to contrib, and then remove
FieldSelector from core.
> Add narrow API for loading stored fields, to replace FieldSelector
> ------------------------------------------------------------------
>
> Key: LUCENE-3309
> URL: https://issues.apache.org/jira/browse/LUCENE-3309
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Fix For: Field Type branch
>
> Attachments: LUCENE-3309.patch
>
>
> I think we should "invert" the FieldSelector API, with a "push" API
> whereby FieldsReader invokes this API once per field in the document
> being visited.
> Implementations of the API can then do arbitrary things like save away
> the field's size, load the field, clone the IndexInput for later lazy
> loading, etc.
> This very thin API would be a mirror image of the very thin index time
> API we now have (IndexableField) and, importantly, it would have no
> dependence on our "user space" Document/Field/FieldType impl, so apps
> are free to do something totally custom.
> After we have this, we should build the "sugar" API that rebuilds a
> Document instance (ie IR.document(int docID)) on top of this new thin
> API. This'll also be a good test that the API is sufficient.
> Relevant discussions from IRC this morning at
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-07-13#l76
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3309) Add narrow API for loading stored
fields, to replace FieldSelector
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-3309.
----------------------------------------
Resolution: Fixed
> Add narrow API for loading stored fields, to replace FieldSelector
> ------------------------------------------------------------------
>
> Key: LUCENE-3309
> URL: https://issues.apache.org/jira/browse/LUCENE-3309
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Fix For: Field Type branch
>
> Attachments: LUCENE-3309.patch, LUCENE-3309.patch, LUCENE-3309.patch
>
>
> I think we should "invert" the FieldSelector API, with a "push" API
> whereby FieldsReader invokes this API once per field in the document
> being visited.
> Implementations of the API can then do arbitrary things like save away
> the field's size, load the field, clone the IndexInput for later lazy
> loading, etc.
> This very thin API would be a mirror image of the very thin index time
> API we now have (IndexableField) and, importantly, it would have no
> dependence on our "user space" Document/Field/FieldType impl, so apps
> are free to do something totally custom.
> After we have this, we should build the "sugar" API that rebuilds a
> Document instance (ie IR.document(int docID)) on top of this new thin
> API. This'll also be a good test that the API is sufficient.
> Relevant discussions from IRC this morning at
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-07-13#l76
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3309) Add narrow API for loading stored
fields, to replace FieldSelector
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065223#comment-13065223 ]
Michael McCandless commented on LUCENE-3309:
--------------------------------------------
bq. ah man I am to late
No you're not! It's just software ;) The iterations go on and on and on...
bq. I only looked at the StoredFieldVisitor briefly, what I always missed is a String... fieldsToLoad ctor for convenience which simply passes an array that is used to create a set
You mean DocumentStoredFieldVisitor, right? Ie the default/sugar visitor that makes a Document. I agree we should add a sugar ctor taking String... I'll do that.
bq. I also don't understand why the set is null in the default ctors, can't we use the empty set here.
Hmm but the default ctor means all fields are accepted (empty set would mean none).
bq. If we want to prevent the method call if no fields should be loaded there should be a LoadNoFieldsVisitor I think, no?
Wait, shouldn't the app just not call IR.document() if it wants no fields?
> Add narrow API for loading stored fields, to replace FieldSelector
> ------------------------------------------------------------------
>
> Key: LUCENE-3309
> URL: https://issues.apache.org/jira/browse/LUCENE-3309
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Fix For: Field Type branch
>
> Attachments: LUCENE-3309.patch, LUCENE-3309.patch, LUCENE-3309.patch
>
>
> I think we should "invert" the FieldSelector API, with a "push" API
> whereby FieldsReader invokes this API once per field in the document
> being visited.
> Implementations of the API can then do arbitrary things like save away
> the field's size, load the field, clone the IndexInput for later lazy
> loading, etc.
> This very thin API would be a mirror image of the very thin index time
> API we now have (IndexableField) and, importantly, it would have no
> dependence on our "user space" Document/Field/FieldType impl, so apps
> are free to do something totally custom.
> After we have this, we should build the "sugar" API that rebuilds a
> Document instance (ie IR.document(int docID)) on top of this new thin
> API. This'll also be a good test that the API is sufficient.
> Relevant discussions from IRC this morning at
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-07-13#l76
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3309) Add narrow API for loading stored
fields, to replace FieldSelector
Posted by "Chris Male (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065046#comment-13065046 ]
Chris Male commented on LUCENE-3309:
------------------------------------
Hey Mike,
Wow you're fast! I love the ideas in this. Just a few things:
- Applied your patch and theres something not quite right. It doesn't contain FieldSelectorVisitor and FieldSelector hasn't been moved to contrib/misc (in fact, there are imports of document.FieldSelector when its actually in document2.FieldSelector).
- Can we rename FieldVisitor to StoredFieldVisitor? FieldVisitor sounds really general and like it visits all fields, not just stored ones.
- What do you think about popping FieldVisitor and the standard impl out of IndexReader? I'm envisaging situations where people may want to extend the standard impl in some way.
- I saw a few javadoc errors floating round but they may just be intellij lying to me.
> Add narrow API for loading stored fields, to replace FieldSelector
> ------------------------------------------------------------------
>
> Key: LUCENE-3309
> URL: https://issues.apache.org/jira/browse/LUCENE-3309
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Fix For: Field Type branch
>
> Attachments: LUCENE-3309.patch, LUCENE-3309.patch
>
>
> I think we should "invert" the FieldSelector API, with a "push" API
> whereby FieldsReader invokes this API once per field in the document
> being visited.
> Implementations of the API can then do arbitrary things like save away
> the field's size, load the field, clone the IndexInput for later lazy
> loading, etc.
> This very thin API would be a mirror image of the very thin index time
> API we now have (IndexableField) and, importantly, it would have no
> dependence on our "user space" Document/Field/FieldType impl, so apps
> are free to do something totally custom.
> After we have this, we should build the "sugar" API that rebuilds a
> Document instance (ie IR.document(int docID)) on top of this new thin
> API. This'll also be a good test that the API is sufficient.
> Relevant discussions from IRC this morning at
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-07-13#l76
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3309) Add narrow API for loading stored
fields, to replace FieldSelector
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065289#comment-13065289 ]
Simon Willnauer commented on LUCENE-3309:
-----------------------------------------
bq. Wait, shouldn't the app just not call IR.document() if it wants no fields?
nevermind I was confused about the null set.
simon
> Add narrow API for loading stored fields, to replace FieldSelector
> ------------------------------------------------------------------
>
> Key: LUCENE-3309
> URL: https://issues.apache.org/jira/browse/LUCENE-3309
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Fix For: Field Type branch
>
> Attachments: LUCENE-3309.patch, LUCENE-3309.patch, LUCENE-3309.patch
>
>
> I think we should "invert" the FieldSelector API, with a "push" API
> whereby FieldsReader invokes this API once per field in the document
> being visited.
> Implementations of the API can then do arbitrary things like save away
> the field's size, load the field, clone the IndexInput for later lazy
> loading, etc.
> This very thin API would be a mirror image of the very thin index time
> API we now have (IndexableField) and, importantly, it would have no
> dependence on our "user space" Document/Field/FieldType impl, so apps
> are free to do something totally custom.
> After we have this, we should build the "sugar" API that rebuilds a
> Document instance (ie IR.document(int docID)) on top of this new thin
> API. This'll also be a good test that the API is sufficient.
> Relevant discussions from IRC this morning at
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-07-13#l76
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3309) Add narrow API for loading stored
fields, to replace FieldSelector
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3309:
---------------------------------------
Attachment: LUCENE-3309.patch
Thanks Chris -- great feedback! I agree with all your suggestions...
New patch:
* Removed oal.documen2.FieldSelector*
* Renamed & moved to standalone StoredFieldVisitor.java
* Made the default/sugar visitor first class, as
DocumentStoredFieldVisitor.java, also allowing it to optionally
accept Set<String> fieldsToLoad.
* Fix javadoc errors I think I caused, but we [separately] have
pre-existing javadoc errors that need fixing (Nikola: can you
address these? Thanks!).
Sorry, my changes have some svn mvs, and my last patch was from "svn
diff". New patch should be fully applyable... (to field type
branch).
> Add narrow API for loading stored fields, to replace FieldSelector
> ------------------------------------------------------------------
>
> Key: LUCENE-3309
> URL: https://issues.apache.org/jira/browse/LUCENE-3309
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Fix For: Field Type branch
>
> Attachments: LUCENE-3309.patch, LUCENE-3309.patch, LUCENE-3309.patch
>
>
> I think we should "invert" the FieldSelector API, with a "push" API
> whereby FieldsReader invokes this API once per field in the document
> being visited.
> Implementations of the API can then do arbitrary things like save away
> the field's size, load the field, clone the IndexInput for later lazy
> loading, etc.
> This very thin API would be a mirror image of the very thin index time
> API we now have (IndexableField) and, importantly, it would have no
> dependence on our "user space" Document/Field/FieldType impl, so apps
> are free to do something totally custom.
> After we have this, we should build the "sugar" API that rebuilds a
> Document instance (ie IR.document(int docID)) on top of this new thin
> API. This'll also be a good test that the API is sufficient.
> Relevant discussions from IRC this morning at
> http://colabti.org/irclogger/irclogger_log/lucene-dev?date=2011-07-13#l76
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org