You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Venkat Kranthi Chalasani <kr...@icloud.com.INVALID> on 2019/03/02 09:12:16 UTC
Fetching 1000 documents taking around 30ms
Hi,
I have an index of ~4M documents. My queries are running in 1-2ms but fetching the top hits (~1000 documents) takes around 30ms. Is this expected?
If it is, I was wondering if maintaining an application cache with docId as a key is ok. I understand docIds are ephemeral as documents are added/deleted but in our usage of lucene, we don’t add/delete documents.
thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Fetching 1000 documents taking around 30ms
Posted by Venkat Kranthi Chalasani <kr...@icloud.com.INVALID>.
Thank you for the explanation. After storing the numeric field I need as a docValue, the fetch time came down to 2ms.
thanks
-kranthi
> On Mar 2, 2019, at 9:54 AM, Erick Erickson <er...@gmail.com> wrote:
>
> “Is this expected”
>
> Yes. For each document, if there is any field with stored=true that does _not_ have docValues=true or is flagged as useDocValuesAsStored=false, there is
> 1> a disk seek to read the stored data from the fdt file
> 2> decompression of the data read in <1>, 16K block minimum.
>
> So getting this all in 30 ms for 1,000 docs isn’t bad at all.
>
> If (and only if) _all_ the values you ask for are docValues=true and useDocValuesAsStored=true then all the values will be returned from the in-memory docValues data.
>
> Best,
> Erick
>
>
>
>> On Mar 2, 2019, at 1:12 AM, Venkat Kranthi Chalasani <kr...@icloud.com.INVALID> wrote:
>>
>> Hi,
>>
>> I have an index of ~4M documents. My queries are running in 1-2ms but fetching the top hits (~1000 documents) takes around 30ms. Is this expected?
>>
>> If it is, I was wondering if maintaining an application cache with docId as a key is ok. I understand docIds are ephemeral as documents are added/deleted but in our usage of lucene, we don’t add/delete documents.
>>
>> thanks
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Fetching 1000 documents taking around 30ms
Posted by Erick Erickson <er...@gmail.com>.
“Is this expected”
Yes. For each document, if there is any field with stored=true that does _not_ have docValues=true or is flagged as useDocValuesAsStored=false, there is
1> a disk seek to read the stored data from the fdt file
2> decompression of the data read in <1>, 16K block minimum.
So getting this all in 30 ms for 1,000 docs isn’t bad at all.
If (and only if) _all_ the values you ask for are docValues=true and useDocValuesAsStored=true then all the values will be returned from the in-memory docValues data.
Best,
Erick
> On Mar 2, 2019, at 1:12 AM, Venkat Kranthi Chalasani <kr...@icloud.com.INVALID> wrote:
>
> Hi,
>
> I have an index of ~4M documents. My queries are running in 1-2ms but fetching the top hits (~1000 documents) takes around 30ms. Is this expected?
>
> If it is, I was wondering if maintaining an application cache with docId as a key is ok. I understand docIds are ephemeral as documents are added/deleted but in our usage of lucene, we don’t add/delete documents.
>
> thanks
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org