You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by eksdev <ek...@googlemail.com> on 2012/11/08 08:56:44 UTC

Compressed stored fields and multiGet(sorted luceneId[])?

Just a theoretical question, would it make sense to add some sort of StoredDocument[] bulkGet(int[] docId) to fetch multiple stored documents in one go? 

The reasoning behind is that now with compressed blocks random-access gets more expensive, and in some cases  a user  needs to fetch more documents in one go. If it happens that more documents come from one block it is a win. I would also assume, even without compression , bulk access on sorted docIds cold be a win (sequential access)?

Does that make sense, is it doable? Or even worse, does it already exist :)

By the way, I am impressed how well compression does, even on really short stored documents, approx. 150b  we observe 35% reduction. Fetching 1000 short documents on fully cached index  is observably slower (2-3 times), but as soon as you memory gets low, compression wins quickly. Did not test it thoroughly, but looks good so far. Great job!


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Compressed stored fields and multiGet(sorted luceneId[])?

Posted by Adrien Grand <jp...@gmail.com>.
Hi!

I'll try to complete what Simon and Robert said:

On Thu, Nov 8, 2012 at 8:56 AM, eksdev <ek...@googlemail.com> wrote:

> Just a theoretical question, would it make sense to add some sort of
> StoredDocument[] bulkGet(int[] docId) to fetch multiple stored documents in
> one go?
>
> The reasoning behind is that now with compressed blocks random-access gets
> more expensive, and in some cases  a user  needs to fetch more documents in
> one go. If it happens that more documents come from one block it is a win.
> I would also assume, even without compression , bulk access on sorted
> docIds cold be a win (sequential access)?
>
> Does that make sense, is it doable? Or even worse, does it already exist :)
>

Even with small documents (100 bytes, 160 docs per chunk) and a small index
(100K docs), there would still be 625 chunks, so the probability of two
documents of the same results page being in the same chunk is very low. So
I think we should not optimize for this case.

However, CompressingStoredFieldsFormat implements efficient sequential
iteration internally in order to improve merging performance: when merging
a segment, every chunk gets decompressed only once.

By the way, I am impressed how well compression does, even on really short
> stored documents, approx. 150b  we observe 35% reduction. Fetching 1000
> short documents on fully cached index  is observably slower (2-3 times),
> but as soon as you memory gets low, compression wins quickly.
>

Awesome! Thank you for trying it!

-- 
Adrien

Re: Compressed stored fields and multiGet(sorted luceneId[])?

Posted by eksdev <ek...@googlemail.com>.
On Nov 8, 2012, at 11:30 AM, Robert Muir <rc...@gmail.com> wrote:

Thanks everybody for response, and  much more of the same for the great project


> Why are you retrieving thousands of stored fields?


 I do not  think it is all that rare that people  actually do something with information  but display summaries?  
Clustering in solr does exactly that,  online record linkage follows exactly the same pattern. 
 
A pattern "fetch thousands of candidates and run some heavy processing on them" is surely not  a typical "web search engine"  usage, but  philosophically,  a model:
a) search data 
b) do something with it
c) deliver 
is not that strange?

You say, b) should not be done using stored fields, ok I trust you, but going to database/nosql/anything  is even slower. What approach would you recommend? 


"the probability of two documents of the same results page being in the same chunk is very low."

Adrian, Robert, this is 100% correct, no objection there.   
In this particular case we are using locality of reference heavily.  We simply sort the data and reindex from time to time. You have to be lucky to be able to sort the documents, but  we do not use lucene for big chunks of text, rather for almost fully structured data and we know how to sort this data to preserve locality of reference… Also a bit unusual, but  I do not think all that rare scenario. 
Sorting data (where possible) was a great optimisation tip for many applications, even before compression.


"really you should roll your own codec for this and specialise."

Yes, already started thinking about it, but we  will first try to play with chunk size to see if we can achieve the goal without own codec …



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Compressed stored fields and multiGet(sorted luceneId[])?

Posted by Simon Willnauer <si...@gmail.com>.
On Thu, Nov 8, 2012 at 11:30 AM, Robert Muir <rc...@gmail.com> wrote:
> Why are you retrieving thousands of stored fields?

really you should roll your own codec for this and specialize. This is
a very unique usecase.

simon
>
> I don't think we should add such an API: stored fields shouldn't be
> used like that, only for things like summary results, so the
> possibility of two documents being in the same block is not so high
> anyway.
>
> And i think the api to stay the way it is (simple visitor) to
> encourage the fact that people shouldnt use stored fields for
> "processing".
>
> On Thu, Nov 8, 2012 at 2:56 AM, eksdev <ek...@googlemail.com> wrote:
>> Just a theoretical question, would it make sense to add some sort of StoredDocument[] bulkGet(int[] docId) to fetch multiple stored documents in one go?
>>
>> The reasoning behind is that now with compressed blocks random-access gets more expensive, and in some cases  a user  needs to fetch more documents in one go. If it happens that more documents come from one block it is a win. I would also assume, even without compression , bulk access on sorted docIds cold be a win (sequential access)?
>>
>> Does that make sense, is it doable? Or even worse, does it already exist :)
>>
>> By the way, I am impressed how well compression does, even on really short stored documents, approx. 150b  we observe 35% reduction. Fetching 1000 short documents on fully cached index  is observably slower (2-3 times), but as soon as you memory gets low, compression wins quickly. Did not test it thoroughly, but looks good so far. Great job!
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: Compressed stored fields and multiGet(sorted luceneId[])?

Posted by Robert Muir <rc...@gmail.com>.
Why are you retrieving thousands of stored fields?

I don't think we should add such an API: stored fields shouldn't be
used like that, only for things like summary results, so the
possibility of two documents being in the same block is not so high
anyway.

And i think the api to stay the way it is (simple visitor) to
encourage the fact that people shouldnt use stored fields for
"processing".

On Thu, Nov 8, 2012 at 2:56 AM, eksdev <ek...@googlemail.com> wrote:
> Just a theoretical question, would it make sense to add some sort of StoredDocument[] bulkGet(int[] docId) to fetch multiple stored documents in one go?
>
> The reasoning behind is that now with compressed blocks random-access gets more expensive, and in some cases  a user  needs to fetch more documents in one go. If it happens that more documents come from one block it is a win. I would also assume, even without compression , bulk access on sorted docIds cold be a win (sequential access)?
>
> Does that make sense, is it doable? Or even worse, does it already exist :)
>
> By the way, I am impressed how well compression does, even on really short stored documents, approx. 150b  we observe 35% reduction. Fetching 1000 short documents on fully cached index  is observably slower (2-3 times), but as soon as you memory gets low, compression wins quickly. Did not test it thoroughly, but looks good so far. Great job!
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org