You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by Ryan McKinley <ry...@gmail.com> on 2007/01/11 07:29:26 UTC
adding "modes" to the command
How do you all feel about adding various "modes" to the <add> command?
Something like:
mode="add or replace document" (default, the current behavior)
mode="add or replace fields"
mode="add fields"
mode="add distinct fields"
The reason i ask is that i would like to frequently update a few
fields without having to know anything about the other fields.
Ideally it would be from a command like this:
<addFromSQL
mode="add or replace fields"
connection="jdbc:mysql://localhost/nblmc?username=xxx&password=xxx"
driver="com.mysql.jdbc.Driver"
multifieldSeperator="\n" >
SELECT * FROM my_stats_table
</addFromSQL>
(I'll save the discussion of <addFromSQL ...> for the next email)
To give you examples of how i think these "modes" would behave,
consider you have a database with the following document:
<doc>
<field name="id">XYZ</field>
<field name="count">10</field>
<field name="cat">A</field>
<field name="cat">B</field>
<field name="cat">C</field>
</doc>
------------------------
AFTER:
<add mode="add or replace fields">
<doc>
<field name="id">XYZ</field>
<field name="cat">C</field>
<field name="cat">D</field>
</doc>
</add>
You would have:
<doc>
<field name="id">XYZ</field>
<field name="count">10</field>
<field name="cat">C</field>
<field name="cat">D</field>
</doc>
------------------
AFTER:
<add mode="add fields">
<doc>
<field name="id">XYZ</field>
<field name="cat">C</field>
<field name="cat">D</field>
</doc>
</add>
You would have:
<doc>
<field name="id">XYZ</field>
<field name="count">10</field>
<field name="cat">A</field>
<field name="cat">B</field>
<field name="cat">C</field>
<field name="cat">C</field>
<field name="cat">D</field>
</doc>
-------------------------
AFTER:
<add mode="add distinct fields">
<doc>
<field name="id">XYZ</field>
<field name="cat">C</field>
<field name="cat">D</field>
</doc>
</add>
You would have:
<doc>
<field name="id">XYZ</field>
<field name="count">10</field>
<field name="cat">A</field>
<field name="cat">B</field>
<field name="cat">C</field>
<field name="cat">D</field>
</doc>
I *think* it should even have the same thing if it were given:
<add mode="add distinct fields">
<doc>
<field name="id">XYZ</field>
<field name="cat">C</field>
<field name="cat">C</field>
<field name="cat">C</field>
<field name="cat">C</field>
<field name="cat">D</field>
</doc>
</add>
-------------------
Although I am suggesting this for my immediate need (update from SQL),
it seems like this would also be useful for anyone building a
javascript 'tagging' library. The client could use the "add distinct
fields" mode without worrying about the rest of the document.
Essentially, this would ask the server to load and merge documents.
If it were added, i think it would go in
DirectUpdateHandler.addDoc(AddupdateCommand) [line 312] before
checking dupes and overwrite stuff. To accuratly load the 'current'
document, it would have to first check the updateHandler.searcher (in
case the document is in the updating index) then check the 'real'
index. BUT, checking the DirectUpdateHandler.searcher every time you
add a document would not be great because the searcher is closed each
time.
Thoughts?
ryan
Re: adding "modes" to the command
Posted by Bertrand Delacretaz <bd...@apache.org>.
On 1/11/07, Ryan McKinley <ry...@gmail.com> wrote:
> How do you all feel about adding various "modes" to the <add> command?
>
> Something like:
> mode="add or replace document" (default, the current behavior)
> mode="add or replace fields"
> mode="add fields"
> mode="add distinct fields"...
Although this is useful as you explain, I like the current simplicity
of the Solr HTTP/XML interface very much.
The more options we add, the harder it becomes to understand and test
the interface.
So, IMHO, it would be good for this functionality to be provided in a
plugin, disabled by default. IIUC the modifications the the Solr core
are minimal, these can included in the core, but (again IMHO, this is
debatable for sure) the public interface to this should be provided by
a "special" plugin.
-Bertrand
Re: adding "modes" to the command
Posted by Yonik Seeley <yo...@apache.org>.
On 1/11/07, J.J. Larrea <jj...@panix.com> wrote:
> I'll observe that Luke has a "Reconstruct and Edit" function which displays the indexed values for each field for the selected Document when stored values aren't available... it iterates the entire inverted index and intersects each term position vector with the target Document ID via TermPositions.skipTo(id).
Right, and that's *very* slow for a large index. IMO, it would be
better to add a restriction that if you want the ability to update
fields on an existing document, then all the fields must be stored.
One might be able to do some magic with ParallelIndex, but that too
comes with it's own set of problems, namely keeping the indicies in
"sync".
-Yonik
Re: adding "modes" to the command
Posted by "J.J. Larrea" <jj...@panix.com>.
At 6:43 AM -0500 1/11/07, Erik Hatcher wrote:
>If all fields are stored, the implementation could simply pull them all into memory on the Solr side and add the document as if it had been sent entirely by the client. But, what happens when for un-stored fields?
I'll observe that Luke has a "Reconstruct and Edit" function which displays the indexed values for each field for the selected Document when stored values aren't available... it iterates the entire inverted index and intersects each term position vector with the target Document ID via TermPositions.skipTo(id).
While that would be too slow to do on a per-update basis, it might be feasible for an update function if it cached a list of partially defined Documents and only at the end (at closing or whenever the list grew past a defined maximum) did a bulk intersection to find indexed values which are not overridden with new values, with just a single traversal of the index in Term then updated DocID order. Once done the reconstructed Documents could be added and the prior versions deleted.
The roadblocks come up when re-adding the indexed values to the index: while the updater can create a new untokenized unstored Field for each indexed value so it is literally re-added, in that case there is no way to externally specify the position offset to match the original. DocumentWriter and the classes it relies on are package-private and final, so no way to interpose there. But an effective hack might be to set the reconstructed Fields to tokenized but specify for those fields a special Analyzer which acts like Keyword Analyzer but looks up the position offset in a table created by the update mechanism and returns it with the token. A little convoluted but probably doable if someone had the time and inclination.
- J.J.
Re: adding "modes" to the command
Posted by Ryan McKinley <ry...@gmail.com>.
On 1/11/07, Erik Hatcher <er...@ehatchersolutions.com> wrote:
>
> On Jan 11, 2007, at 1:29 AM, Ryan McKinley wrote:
> > mode="add or replace fields"
> > mode="add fields"
> > mode="add distinct fields"
> >
> > The reason i ask is that i would like to frequently update a few
> > fields without having to know anything about the other fields.
>
> If you can implement a way to do this efficiently with Lucene, you
> will be my (and many others) hero!
>
I'm still new to this, so *please* don't mistake my ambition for hubris!
I understand lucene needs to add all document fields at the same time.
I don't have any magic idea to change that. BUT it seems like solr
is a good place to offer a syntax that lets users ignore feel like
they are just updating a field rather then loading all fields and
reindexing.
> In Lucene to update a document the operation is really a delete
> followed by an add. You will need to add the complete document as
> there is no such "update only a field" semantics in Lucene.
>
> If all fields are stored, the implementation could simply pull them
> all into memory on the Solr side and add the document as if it had
> been sent entirely by the client. But, what happens when for un-
> stored fields?
>
for the unstored fields, is it possible to read the tokens (and all
info) and then write them back directly? Does lucene let you do this
directly? or do would i need to write a Tokenizer that takes the old
list of tokens and re-tokenizes them?
I guess this would require a slightly different DocumentBuilder for
'updated' fields where you would skip the analyzers defined by the
schemaField.
Re: adding "modes" to the command
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jan 11, 2007, at 1:29 AM, Ryan McKinley wrote:
> mode="add or replace fields"
> mode="add fields"
> mode="add distinct fields"
>
> The reason i ask is that i would like to frequently update a few
> fields without having to know anything about the other fields.
If you can implement a way to do this efficiently with Lucene, you
will be my (and many others) hero!
In Lucene to update a document the operation is really a delete
followed by an add. You will need to add the complete document as
there is no such "update only a field" semantics in Lucene.
If all fields are stored, the implementation could simply pull them
all into memory on the Solr side and add the document as if it had
been sent entirely by the client. But, what happens when for un-
stored fields?
Erik