You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Praveen Peddi <pp...@contextmedia.com> on 2004/12/08 23:53:26 UTC
partial updating of lucene
Hi all,
I have a question about updating the lucene document. I know that there is no API to do that now. So this is what I am doing in order to update the document with the field "title".
1) Get the document from lucene index
2) Remove a field called "title" and add the same field with a modified value
3) Remove the docment (based on one of our field) using Reader and then close the Reader.
4) Add the document that is obtained in 1 and modified in 2.
I am not sure if this is the right way of doing it but I am having problems searching for that document after updating it. The problem is only with the un stored fields.
For example, I search as "description:boy" where description is a unstored, indexed, tokenized field in the document. I find 1 document. Now I update the document the document's title as descripbed above and repeat the same search "description:boy" and now I don't find any results. I have not touched the field "description" at all. I just updated the field title.
Is this an expected behaviour? If not, is it a bug.
If I change the field description as stored, indexed and tokenized, the search works fine before and after updating.
Praveen
**************************************************************
Praveen Peddi
Sr Software Engg, Context Media, Inc.
email:ppeddi@contextmedia.com
Tel: 401.854.3475
Fax: 401.861.3596
web: http://www.contextmedia.com
**************************************************************
Context Media- "The Leader in Enterprise Content Integration"
Re: partial updating of lucene
Posted by Praveen Peddi <pp...@contextmedia.com>.
But I don't need anything that Limo or Luke is doing, if all my fields are
stored in the index (isStored() will be true for all fields). right?
Praveen
----- Original Message -----
From: "Luke Francl" <lu...@stellent.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Thursday, December 09, 2004 11:30 AM
Subject: Re: partial updating of lucene
> On Thu, 2004-12-09 at 09:00, Erik Hatcher wrote:
>
>> Have a look at the tool Luke (Google for "luke lucene" :) and see how
>> it does its "Reconstruct and Edit" facility. It is possible, though
>> potentially lossy, to reconstruct a document and add it again.
>
> Or look at LIMO's implementation of that feature, which to my eyes is a
> little easier to read (of course that's probably because I wrote it...
> ;):
>
> http://cvs.sourceforge.net/viewcvs.py/limo/limo/src/net/sourceforge/limo/LimoUtils.java?rev=1.6&view=markup
>
> (check out LimoUtils.reconstructDocument())
>
> However, if you're doing analysis on your text to remove stopwords and
> stuff like that, this WILL be lossy. I consider it more of an aid for
> debugging than a way to re-index documents, though I suppose it would
> work for that as well. However, I believe the process would be highly
> resource intensive so I wouldn't recommend it.
>
> The better solution is to add a stored keyword field that stores the
> location of your document, and then re-index it from the source.
>
> Regards,
> Luke Francl
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: partial updating of lucene
Posted by Luke Francl <lu...@stellent.com>.
On Thu, 2004-12-09 at 09:00, Erik Hatcher wrote:
> Have a look at the tool Luke (Google for "luke lucene" :) and see how
> it does its "Reconstruct and Edit" facility. It is possible, though
> potentially lossy, to reconstruct a document and add it again.
Or look at LIMO's implementation of that feature, which to my eyes is a
little easier to read (of course that's probably because I wrote it...
;):
http://cvs.sourceforge.net/viewcvs.py/limo/limo/src/net/sourceforge/limo/LimoUtils.java?rev=1.6&view=markup
(check out LimoUtils.reconstructDocument())
However, if you're doing analysis on your text to remove stopwords and
stuff like that, this WILL be lossy. I consider it more of an aid for
debugging than a way to re-index documents, though I suppose it would
work for that as well. However, I believe the process would be highly
resource intensive so I wouldn't recommend it.
The better solution is to add a stored keyword field that stores the
location of your document, and then re-index it from the source.
Regards,
Luke Francl
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: partial updating of lucene
Posted by Praveen Peddi <pp...@contextmedia.com>.
If I store all the fields I am indexing, is it safe to get the document,
update a fields and add it again to the search index? I do not want to lose
anything and I want to make sure that document is same before and after
updating (execpt for the updated fields).
Praveen
----- Original Message -----
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Thursday, December 09, 2004 10:00 AM
Subject: Re: partial updating of lucene
> On Dec 9, 2004, at 9:48 AM, Praveen Peddi wrote:
>> But when I am searching, it only searches in the index. Stored fields are
>> only used to display the results, not to search. Why would it lose the
>> terms in the index when I retrieve the document?
>
>> First solution is not possible (I can't create a new document) since I
>> only have modified fields.
>>
>> When I get a document, doesn't the fields have indexed terms along with
>> it? Is there no way to get a full document (along with indexed terms) and
>> clone it and add it to the index?
>>
>> Well is there anyway I ca update a document with just one field (because
>> I only have data for that one field)?
>
> A Document only carries along its *stored* fields. Fields that are
> indexed, but not stored, are not retrievable from Document.
>
> Have a look at the tool Luke (Google for "luke lucene" :) and see how it
> does its "Reconstruct and Edit" facility. It is possible, though
> potentially lossy, to reconstruct a document and add it again.
>
> Erik
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: partial updating of lucene
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Dec 9, 2004, at 9:48 AM, Praveen Peddi wrote:
> But when I am searching, it only searches in the index. Stored fields
> are only used to display the results, not to search. Why would it lose
> the terms in the index when I retrieve the document?
> First solution is not possible (I can't create a new document) since I
> only have modified fields.
>
> When I get a document, doesn't the fields have indexed terms along
> with it? Is there no way to get a full document (along with indexed
> terms) and clone it and add it to the index?
>
> Well is there anyway I ca update a document with just one field
> (because I only have data for that one field)?
A Document only carries along its *stored* fields. Fields that are
indexed, but not stored, are not retrievable from Document.
Have a look at the tool Luke (Google for "luke lucene" :) and see how
it does its "Reconstruct and Edit" facility. It is possible, though
potentially lossy, to reconstruct a document and add it again.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: partial updating of lucene
Posted by Praveen Peddi <pp...@contextmedia.com>.
But when I am searching, it only searches in the index. Stored fields are
only used to display the results, not to search. Why would it lose the terms
in the index when I retrieve the document?
First solution is not possible (I can't create a new document) since I only
have modified fields.
When I get a document, doesn't the fields have indexed terms along with it?
Is there no way to get a full document (along with indexed terms) and clone
it and add it to the index?
Well is there anyway I ca update a document with just one field (because I
only have data for that one field)?
Praveen
----- Original Message -----
From: "Justin Swanhart" <gr...@gmail.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Wednesday, December 08, 2004 5:59 PM
Subject: Re: partial updating of lucene
> You unstored fields were not stored in the index, only their terms
> were stored. When you get the document from the index and modify it,
> those terms are lost when you add the document again.
>
> You can either simply create a new document and populate all the
> fields and add that document to the index, or you can add the unstored
> fields to the document retrieved in step 1.
>
>
> On Wed, 8 Dec 2004 17:53:26 -0500, Praveen Peddi
> <pp...@contextmedia.com> wrote:
>> Hi all,
>> I have a question about updating the lucene document. I know that there
>> is no API to do that now. So this is what I am doing in order to update
>> the document with the field "title".
>>
>> 1) Get the document from lucene index
>> 2) Remove a field called "title" and add the same field with a modified
>> value
>> 3) Remove the docment (based on one of our field) using Reader and then
>> close the Reader.
>> 4) Add the document that is obtained in 1 and modified in 2.
>>
>> I am not sure if this is the right way of doing it but I am having
>> problems searching for that document after updating it. The problem is
>> only with the un stored fields.
>>
>> For example, I search as "description:boy" where description is a
>> unstored, indexed, tokenized field in the document. I find 1 document.
>> Now I update the document the document's title as descripbed above and
>> repeat the same search "description:boy" and now I don't find any
>> results. I have not touched the field "description" at all. I just
>> updated the field title.
>>
>> Is this an expected behaviour? If not, is it a bug.
>>
>> If I change the field description as stored, indexed and tokenized, the
>> search works fine before and after updating.
>>
>> Praveen
>> **************************************************************
>> Praveen Peddi
>> Sr Software Engg, Context Media, Inc.
>> email:ppeddi@contextmedia.com
>> Tel: 401.854.3475
>> Fax: 401.861.3596
>> web: http://www.contextmedia.com
>> **************************************************************
>> Context Media- "The Leader in Enterprise Content Integration"
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: partial updating of lucene
Posted by Justin Swanhart <gr...@gmail.com>.
You unstored fields were not stored in the index, only their terms
were stored. When you get the document from the index and modify it,
those terms are lost when you add the document again.
You can either simply create a new document and populate all the
fields and add that document to the index, or you can add the unstored
fields to the document retrieved in step 1.
On Wed, 8 Dec 2004 17:53:26 -0500, Praveen Peddi
<pp...@contextmedia.com> wrote:
> Hi all,
> I have a question about updating the lucene document. I know that there is no API to do that now. So this is what I am doing in order to update the document with the field "title".
>
> 1) Get the document from lucene index
> 2) Remove a field called "title" and add the same field with a modified value
> 3) Remove the docment (based on one of our field) using Reader and then close the Reader.
> 4) Add the document that is obtained in 1 and modified in 2.
>
> I am not sure if this is the right way of doing it but I am having problems searching for that document after updating it. The problem is only with the un stored fields.
>
> For example, I search as "description:boy" where description is a unstored, indexed, tokenized field in the document. I find 1 document. Now I update the document the document's title as descripbed above and repeat the same search "description:boy" and now I don't find any results. I have not touched the field "description" at all. I just updated the field title.
>
> Is this an expected behaviour? If not, is it a bug.
>
> If I change the field description as stored, indexed and tokenized, the search works fine before and after updating.
>
> Praveen
> **************************************************************
> Praveen Peddi
> Sr Software Engg, Context Media, Inc.
> email:ppeddi@contextmedia.com
> Tel: 401.854.3475
> Fax: 401.861.3596
> web: http://www.contextmedia.com
> **************************************************************
> Context Media- "The Leader in Enterprise Content Integration"
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org