understanding the need to reindex a document

[Shaya Potter <sp...@gmail.com>]

so there are lots of Qs that are asked about wanting to modify a lucene 
document (i.e. remove fields, add fields....) but are told that one 
needs to reindex.

No one ever answers the technical Q of why this is, and I'm interested 
in that.  presumambly because documents aren't stored as documents or 
even in any form that could be reassembled into a document, but I'm 
interested in what the actual answer is.

thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: understanding the need to reindex a document

[Apostolis Xekoukoulotakis <xe...@gmail.com>]

I cant answer you that. But even if it is feasible, maybe the answer is
that noone really needed it.

2012/10/22 Shaya Potter <sp...@gmail.com>

> I can understand that (i.e. why in place wont work), but the Q would be,
> why one can't read in a document, add() or removeField() that document and
> then updateDocument() that document.
>
>
> On 10/22/2012 03:43 PM, Apostolis Xekoukoulotakis wrote:
>
>> I am not that familiar with Lucene, so my answer may be a bit off.
>>
>> Search on the internet about log structured storage. There you will find
>> why rewriting an entry is better than updating an existing entry.
>>
>> Leveldb/cassandra/bigTable use it. maybe search these terms as well.
>>
>> 2012/10/22 Shaya Potter <sp...@gmail.com>
>>
>>  so there are lots of Qs that are asked about wanting to modify a lucene
>>> document (i.e. remove fields, add fields....) but are told that one needs
>>> to reindex.
>>>
>>> No one ever answers the technical Q of why this is, and I'm interested in
>>> that.  presumambly because documents aren't stored as documents or even
>>> in
>>> any form that could be reassembled into a document, but I'm interested in
>>> what the actual answer is.
>>>
>>> thanks.
>>>
>>> ------------------------------****----------------------------**
>>> --**---------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.****apache.org<
>>> java-user-**unsubscribe@lucene.apache.org<ja...@lucene.apache.org>
>>> >
>>> For additional commands, e-mail: java-user-help@lucene.apache.****org<
>>> java-user-help@lucene.**apache.org <ja...@lucene.apache.org>>
>>>
>>>
>>>
>>
>>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<ja...@lucene.apache.org>
> For additional commands, e-mail: java-user-help@lucene.apache.**org<ja...@lucene.apache.org>
>
>


-- 


Sincerely yours,

     Apostolis Xekoukoulotakis

Re: understanding the need to reindex a document

[Jack Krupansky <ja...@basetechnology.com>]

You can in fact add fields. And you can remove the value of a field for a 
single document. Both can be done by simply re-adding the document with the 
same unique key value. But that won't add that field to other or all 
documents, or remove the value of that field for all other documents.

Re-indexing is usually required when you change the type of a field, its 
attributes, or its analyzer.

The primary difficulty with automatically re-indexing from the index itself 
is that Lucene does not require all fields to be "stored". So, replacing a 
document means you need to have an external source for all non-stored 
fields, or at least non-stored fields which are not copied from another 
field which is stored.

-- Jack Krupansky

-----Original Message----- 
From: Shaya Potter
Sent: Monday, October 22, 2012 3:47 PM
To: java-user@lucene.apache.org
Subject: Re: understanding the need to reindex a document

I can understand that (i.e. why in place wont work), but the Q would be,
why one can't read in a document, add() or removeField() that document
and then updateDocument() that document.

On 10/22/2012 03:43 PM, Apostolis Xekoukoulotakis wrote:
> I am not that familiar with Lucene, so my answer may be a bit off.
>
> Search on the internet about log structured storage. There you will find
> why rewriting an entry is better than updating an existing entry.
>
> Leveldb/cassandra/bigTable use it. maybe search these terms as well.
>
> 2012/10/22 Shaya Potter <sp...@gmail.com>
>
>> so there are lots of Qs that are asked about wanting to modify a lucene
>> document (i.e. remove fields, add fields....) but are told that one needs
>> to reindex.
>>
>> No one ever answers the technical Q of why this is, and I'm interested in
>> that.  presumambly because documents aren't stored as documents or even 
>> in
>> any form that could be reassembled into a document, but I'm interested in
>> what the actual answer is.
>>
>> thanks.
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: 
>> java-user-unsubscribe@lucene.**apache.org<ja...@lucene.apache.org>
>> For additional commands, e-mail: 
>> java-user-help@lucene.apache.**org<ja...@lucene.apache.org>
>>
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: understanding the need to reindex a document

[Rafał Kuć <r....@solr.pl>]

Hello!

This is because how the Lucene inverted index works and the amount of
changes that would have to be done. However there is a work in
progress issue - https://issues.apache.org/jira/browse/LUCENE-3837
There is some technical information there, so you may want to check
it.

-- 
Regards,
 Rafał Kuć
 Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch

> I can understand that (i.e. why in place wont work), but the Q would be,
> why one can't read in a document, add() or removeField() that document
> and then updateDocument() that document.

> On 10/22/2012 03:43 PM, Apostolis Xekoukoulotakis wrote:
>> I am not that familiar with Lucene, so my answer may be a bit off.
>>
>> Search on the internet about log structured storage. There you will find
>> why rewriting an entry is better than updating an existing entry.
>>
>> Leveldb/cassandra/bigTable use it. maybe search these terms as well.
>>
>> 2012/10/22 Shaya Potter <sp...@gmail.com>
>>
>>> so there are lots of Qs that are asked about wanting to modify a lucene
>>> document (i.e. remove fields, add fields....) but are told that one needs
>>> to reindex.
>>>
>>> No one ever answers the technical Q of why this is, and I'm interested in
>>> that.  presumambly because documents aren't stored as documents or even in
>>> any form that could be reassembled into a document, but I'm interested in
>>> what the actual answer is.
>>>
>>> thanks.
>>>
>>> ------------------------------**------------------------------**---------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<ja...@lucene.apache.org>
>>> For additional commands, e-mail: java-user-help@lucene.apache.**org<ja...@lucene.apache.org>
>>>
>>>
>>
>>

> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: understanding the need to reindex a document

[Shaya Potter <sp...@gmail.com>]

I can understand that (i.e. why in place wont work), but the Q would be, 
why one can't read in a document, add() or removeField() that document 
and then updateDocument() that document.

On 10/22/2012 03:43 PM, Apostolis Xekoukoulotakis wrote:
> I am not that familiar with Lucene, so my answer may be a bit off.
>
> Search on the internet about log structured storage. There you will find
> why rewriting an entry is better than updating an existing entry.
>
> Leveldb/cassandra/bigTable use it. maybe search these terms as well.
>
> 2012/10/22 Shaya Potter <sp...@gmail.com>
>
>> so there are lots of Qs that are asked about wanting to modify a lucene
>> document (i.e. remove fields, add fields....) but are told that one needs
>> to reindex.
>>
>> No one ever answers the technical Q of why this is, and I'm interested in
>> that.  presumambly because documents aren't stored as documents or even in
>> any form that could be reassembled into a document, but I'm interested in
>> what the actual answer is.
>>
>> thanks.
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<ja...@lucene.apache.org>
>> For additional commands, e-mail: java-user-help@lucene.apache.**org<ja...@lucene.apache.org>
>>
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: understanding the need to reindex a document

[Apostolis Xekoukoulotakis <xe...@gmail.com>]

I am not that familiar with Lucene, so my answer may be a bit off.

Search on the internet about log structured storage. There you will find
why rewriting an entry is better than updating an existing entry.

Leveldb/cassandra/bigTable use it. maybe search these terms as well.

2012/10/22 Shaya Potter <sp...@gmail.com>

> so there are lots of Qs that are asked about wanting to modify a lucene
> document (i.e. remove fields, add fields....) but are told that one needs
> to reindex.
>
> No one ever answers the technical Q of why this is, and I'm interested in
> that.  presumambly because documents aren't stored as documents or even in
> any form that could be reassembled into a document, but I'm interested in
> what the actual answer is.
>
> thanks.
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<ja...@lucene.apache.org>
> For additional commands, e-mail: java-user-help@lucene.apache.**org<ja...@lucene.apache.org>
>
>


-- 


Sincerely yours,

     Apostolis Xekoukoulotakis