You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by adfel70 <ad...@gmail.com> on 2013/06/12 16:40:22 UTC
Partial update vs full update performance
Hi
As I understand, even if I use partial update, lucene can't really update
documents. Solr will use the stored fields in order to pass the values to
lucene, and a delete,add opeartions will still be performed.
If this is the case is there a performance issue when comparing partial
update to full update?
My documents have dozens of fields, most of them are not stored.
I sometimes need to go through a portion of the documents and modify a
single field.
What I do right now is deleting the portion I want to update, and adding
them with the updated field.
This of course takes a lot of time (I'm talking about ten of millions of
documents).
Should I move to using partial update? will it improve the indexing time at
all? will it improve the indexing time in such extent that I would better be
storing the fields I don't need stored just for the partial update feature?
thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/Partial-update-vs-full-update-performance-tp4069948.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Partial update vs full update performance
Posted by Jack Krupansky <ja...@basetechnology.com>.
Yes, you need to have all the fields stored to do a partial update.
Generally, not storing field values causes all sorts of headaches that far
outweigh the modest benefit in memory savings.
Generally, make everything stored - unless you have specific and VERY
COMPELLING need not to. Back in the early days of Lucene and Solr memory use
was much more compelling. Now, not so much. And even if memory is an issue,
the downside of not storing all values seems much more likely to overwhelm
the benefits.
Sure, there are some apps where you may not want to store much if anything
besides the key (I recall one presentation at Lucene Revolution in San
Diego, and DataStax Enterprise does this because all the data is stored in
Cassandra already), but generally apps would be better off biting the bullet
and throwing memory at the problem.
And DocValues are an alternative if heap space is a critical issue.
2. Large field values are simply a potential issue since they are a lot of
bytes to be retrieved and then re-stored.
-- Jack Krupansky
-----Original Message-----
From: adfel70
Sent: Wednesday, June 12, 2013 11:50 AM
To: solr-user@lucene.apache.org
Subject: Re: Partial update vs full update performance
1. To support partial updates, I must have all the fields stored (most of
which I don't need stored)
Wouldn't I suffer in query perforemnce if I store all these fields?
2. Can you elaborate on the large fields issue?
Why does it matter if the fields are large in the context of partial
updates?
One way or another, lucene will index the field..
Jack Krupansky-2 wrote
> Correct.
>
> Generally, I think most apps will benefit from partial update, especially
> if
> they have a lot of fields. Otherwise, they will have two round trip
> requests
> rather than one. Solr does the reading of existing document values more
> efficiently, under the hood, with no need to format for the response and
> parse the incoming (redundant) values.
>
> OTOH, if the client has all the data anyway (maybe because it wants to
> display the data before update), it may be easier to do a full update.
>
> You could do an actual performance test, but I would suggest that
> (generally) partial update will be more efficient than a full update.
>
> And Lucene can do add and delete rather quickly, so that should not be a
> concern for modest to medium size documents, but clearly would be an issue
> for large and very large documents (hundreds of fields or large field
> values.)
>
> -- Jack Krupansky
>
> -----Original Message-----
> From: adfel70
> Sent: Wednesday, June 12, 2013 10:40 AM
> To:
> solr-user@.apache
> Subject: Partial update vs full update performance
>
> Hi
> As I understand, even if I use partial update, lucene can't really update
> documents. Solr will use the stored fields in order to pass the values to
> lucene, and a delete,add opeartions will still be performed.
>
> If this is the case is there a performance issue when comparing partial
> update to full update?
>
> My documents have dozens of fields, most of them are not stored.
> I sometimes need to go through a portion of the documents and modify a
> single field.
> What I do right now is deleting the portion I want to update, and adding
> them with the updated field.
> This of course takes a lot of time (I'm talking about ten of millions of
> documents).
>
> Should I move to using partial update? will it improve the indexing time
> at
> all? will it improve the indexing time in such extent that I would better
> be
> storing the fields I don't need stored just for the partial update
> feature?
>
> thanks
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Partial-update-vs-full-update-performance-tp4069948.html
> Sent from the Solr - User mailing list archive at Nabble.com.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Partial-update-vs-full-update-performance-tp4069948p4069973.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Partial update vs full update performance
Posted by Shawn Heisey <so...@elyograg.org>.
On 6/12/2013 9:50 AM, adfel70 wrote:
> 1. To support partial updates, I must have all the fields stored (most of
> which I don't need stored)
> Wouldn't I suffer in query perforemnce if I store all these fields?
>
> 2. Can you elaborate on the large fields issue?
> Why does it matter if the fields are large in the context of partial
> updates?
> One way or another, lucene will index the field..
Storing lots of fields does incur performance overhead due to the stored
fields compression that was added in Lucene/Solr 4.1.0. The overhead
can be particularly bad for large fields. You might be able to avoid
some of the problem for the query side by using the fl parameter, but I
could be wrong there, because I think that multiple fields can share the
same compressed block. I'm not very familiar with the Lucene internals.
LUCENE-4995 will improve the situation when 4.4 comes out, but there is
still overhead.
Currently it is not possible to turn compression off. For most typical
use cases, the compression speeds things up, but there are some
situations where it makes things worse.
Thanks,
Shawn
Re: Partial update vs full update performance
Posted by adfel70 <ad...@gmail.com>.
1. To support partial updates, I must have all the fields stored (most of
which I don't need stored)
Wouldn't I suffer in query perforemnce if I store all these fields?
2. Can you elaborate on the large fields issue?
Why does it matter if the fields are large in the context of partial
updates?
One way or another, lucene will index the field..
Jack Krupansky-2 wrote
> Correct.
>
> Generally, I think most apps will benefit from partial update, especially
> if
> they have a lot of fields. Otherwise, they will have two round trip
> requests
> rather than one. Solr does the reading of existing document values more
> efficiently, under the hood, with no need to format for the response and
> parse the incoming (redundant) values.
>
> OTOH, if the client has all the data anyway (maybe because it wants to
> display the data before update), it may be easier to do a full update.
>
> You could do an actual performance test, but I would suggest that
> (generally) partial update will be more efficient than a full update.
>
> And Lucene can do add and delete rather quickly, so that should not be a
> concern for modest to medium size documents, but clearly would be an issue
> for large and very large documents (hundreds of fields or large field
> values.)
>
> -- Jack Krupansky
>
> -----Original Message-----
> From: adfel70
> Sent: Wednesday, June 12, 2013 10:40 AM
> To:
> solr-user@.apache
> Subject: Partial update vs full update performance
>
> Hi
> As I understand, even if I use partial update, lucene can't really update
> documents. Solr will use the stored fields in order to pass the values to
> lucene, and a delete,add opeartions will still be performed.
>
> If this is the case is there a performance issue when comparing partial
> update to full update?
>
> My documents have dozens of fields, most of them are not stored.
> I sometimes need to go through a portion of the documents and modify a
> single field.
> What I do right now is deleting the portion I want to update, and adding
> them with the updated field.
> This of course takes a lot of time (I'm talking about ten of millions of
> documents).
>
> Should I move to using partial update? will it improve the indexing time
> at
> all? will it improve the indexing time in such extent that I would better
> be
> storing the fields I don't need stored just for the partial update
> feature?
>
> thanks
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Partial-update-vs-full-update-performance-tp4069948.html
> Sent from the Solr - User mailing list archive at Nabble.com.
--
View this message in context: http://lucene.472066.n3.nabble.com/Partial-update-vs-full-update-performance-tp4069948p4069973.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Partial update vs full update performance
Posted by adfel70 <ad...@gmail.com>.
Any reason why not index these metadata fields in the same core?
Would I be able to sort, facet with join queries if the joined docs are in
the same core?
Upayavira wrote
> On Wed, Jun 12, 2013, at 04:54 PM, adfel70 wrote:
>> Yes it is.
>> But in my case, these are metadata fields, and I need them to be
>> searchable,
>> facetable, sortable in the context of the main text fields.
>> Will I be able to achieve that if I index them in another core?
>
> Unfortunately, at this point, you can only search on them when they are
> in another core, you cannot facet or sort, meaning join queries won't
> work for you.
>
> Upayavira
--
View this message in context: http://lucene.472066.n3.nabble.com/Partial-update-vs-full-update-performance-tp4069948p4069981.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Partial update vs full update performance
Posted by Upayavira <uv...@odoko.co.uk>.
On Wed, Jun 12, 2013, at 04:54 PM, adfel70 wrote:
> Yes it is.
> But in my case, these are metadata fields, and I need them to be
> searchable,
> facetable, sortable in the context of the main text fields.
> Will I be able to achieve that if I index them in another core?
Unfortunately, at this point, you can only search on them when they are
in another core, you cannot facet or sort, meaning join queries won't
work for you.
Upayavira
Re: Partial update vs full update performance
Posted by adfel70 <ad...@gmail.com>.
Yes it is.
But in my case, these are metadata fields, and I need them to be searchable,
facetable, sortable in the context of the main text fields.
Will I be able to achieve that if I index them in another core?
Upayavira wrote
> My question would be, why are you updating 10m documents? Is it because
> of denormalised fields? E.g. one system I have needs to reindex all data
> for a publication when that publication switches between active and
> inactive.
>
> If this is the case, you can perhaps achieve the same using joins. Store
> the publications, and their status, in another core. Then, to find
> documents for active publications could be:
>
> q=harry potter&fq={!join fromIndex=pubs from=pubID
> to=pubID}status:active
>
> This would find documents containing the terms 'harry potter' which are
> associated with active publications.
>
> Changing the status of a publication would require a single document in
> the 'pubs' core to be changed, rather than re-indexing all documents.
>
> Does this hit what you are trying to achieve?
>
> Upayavira
>
>
> On Wed, Jun 12, 2013, at 03:51 PM, Jack Krupansky wrote:
>> Correct.
>>
>> Generally, I think most apps will benefit from partial update, especially
>> if
>> they have a lot of fields. Otherwise, they will have two round trip
>> requests
>> rather than one. Solr does the reading of existing document values more
>> efficiently, under the hood, with no need to format for the response and
>> parse the incoming (redundant) values.
>>
>> OTOH, if the client has all the data anyway (maybe because it wants to
>> display the data before update), it may be easier to do a full update.
>>
>> You could do an actual performance test, but I would suggest that
>> (generally) partial update will be more efficient than a full update.
>>
>> And Lucene can do add and delete rather quickly, so that should not be a
>> concern for modest to medium size documents, but clearly would be an
>> issue
>> for large and very large documents (hundreds of fields or large field
>> values.)
>>
>> -- Jack Krupansky
>>
>> -----Original Message-----
>> From: adfel70
>> Sent: Wednesday, June 12, 2013 10:40 AM
>> To:
> solr-user@.apache
>> Subject: Partial update vs full update performance
>>
>> Hi
>> As I understand, even if I use partial update, lucene can't really update
>> documents. Solr will use the stored fields in order to pass the values to
>> lucene, and a delete,add opeartions will still be performed.
>>
>> If this is the case is there a performance issue when comparing partial
>> update to full update?
>>
>> My documents have dozens of fields, most of them are not stored.
>> I sometimes need to go through a portion of the documents and modify a
>> single field.
>> What I do right now is deleting the portion I want to update, and adding
>> them with the updated field.
>> This of course takes a lot of time (I'm talking about ten of millions of
>> documents).
>>
>> Should I move to using partial update? will it improve the indexing time
>> at
>> all? will it improve the indexing time in such extent that I would better
>> be
>> storing the fields I don't need stored just for the partial update
>> feature?
>>
>> thanks
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Partial-update-vs-full-update-performance-tp4069948.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
--
View this message in context: http://lucene.472066.n3.nabble.com/Partial-update-vs-full-update-performance-tp4069948p4069974.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Partial update vs full update performance
Posted by Upayavira <uv...@odoko.co.uk>.
My question would be, why are you updating 10m documents? Is it because
of denormalised fields? E.g. one system I have needs to reindex all data
for a publication when that publication switches between active and
inactive.
If this is the case, you can perhaps achieve the same using joins. Store
the publications, and their status, in another core. Then, to find
documents for active publications could be:
q=harry potter&fq={!join fromIndex=pubs from=pubID
to=pubID}status:active
This would find documents containing the terms 'harry potter' which are
associated with active publications.
Changing the status of a publication would require a single document in
the 'pubs' core to be changed, rather than re-indexing all documents.
Does this hit what you are trying to achieve?
Upayavira
On Wed, Jun 12, 2013, at 03:51 PM, Jack Krupansky wrote:
> Correct.
>
> Generally, I think most apps will benefit from partial update, especially
> if
> they have a lot of fields. Otherwise, they will have two round trip
> requests
> rather than one. Solr does the reading of existing document values more
> efficiently, under the hood, with no need to format for the response and
> parse the incoming (redundant) values.
>
> OTOH, if the client has all the data anyway (maybe because it wants to
> display the data before update), it may be easier to do a full update.
>
> You could do an actual performance test, but I would suggest that
> (generally) partial update will be more efficient than a full update.
>
> And Lucene can do add and delete rather quickly, so that should not be a
> concern for modest to medium size documents, but clearly would be an
> issue
> for large and very large documents (hundreds of fields or large field
> values.)
>
> -- Jack Krupansky
>
> -----Original Message-----
> From: adfel70
> Sent: Wednesday, June 12, 2013 10:40 AM
> To: solr-user@lucene.apache.org
> Subject: Partial update vs full update performance
>
> Hi
> As I understand, even if I use partial update, lucene can't really update
> documents. Solr will use the stored fields in order to pass the values to
> lucene, and a delete,add opeartions will still be performed.
>
> If this is the case is there a performance issue when comparing partial
> update to full update?
>
> My documents have dozens of fields, most of them are not stored.
> I sometimes need to go through a portion of the documents and modify a
> single field.
> What I do right now is deleting the portion I want to update, and adding
> them with the updated field.
> This of course takes a lot of time (I'm talking about ten of millions of
> documents).
>
> Should I move to using partial update? will it improve the indexing time
> at
> all? will it improve the indexing time in such extent that I would better
> be
> storing the fields I don't need stored just for the partial update
> feature?
>
> thanks
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Partial-update-vs-full-update-performance-tp4069948.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Partial update vs full update performance
Posted by Jack Krupansky <ja...@basetechnology.com>.
Correct.
Generally, I think most apps will benefit from partial update, especially if
they have a lot of fields. Otherwise, they will have two round trip requests
rather than one. Solr does the reading of existing document values more
efficiently, under the hood, with no need to format for the response and
parse the incoming (redundant) values.
OTOH, if the client has all the data anyway (maybe because it wants to
display the data before update), it may be easier to do a full update.
You could do an actual performance test, but I would suggest that
(generally) partial update will be more efficient than a full update.
And Lucene can do add and delete rather quickly, so that should not be a
concern for modest to medium size documents, but clearly would be an issue
for large and very large documents (hundreds of fields or large field
values.)
-- Jack Krupansky
-----Original Message-----
From: adfel70
Sent: Wednesday, June 12, 2013 10:40 AM
To: solr-user@lucene.apache.org
Subject: Partial update vs full update performance
Hi
As I understand, even if I use partial update, lucene can't really update
documents. Solr will use the stored fields in order to pass the values to
lucene, and a delete,add opeartions will still be performed.
If this is the case is there a performance issue when comparing partial
update to full update?
My documents have dozens of fields, most of them are not stored.
I sometimes need to go through a portion of the documents and modify a
single field.
What I do right now is deleting the portion I want to update, and adding
them with the updated field.
This of course takes a lot of time (I'm talking about ten of millions of
documents).
Should I move to using partial update? will it improve the indexing time at
all? will it improve the indexing time in such extent that I would better be
storing the fields I don't need stored just for the partial update feature?
thanks
--
View this message in context:
http://lucene.472066.n3.nabble.com/Partial-update-vs-full-update-performance-tp4069948.html
Sent from the Solr - User mailing list archive at Nabble.com.