Re: 500 millions document for loop.

[Erick Erickson <er...@gmail.com>]

Actually, this looks like a fine place to use Streaming
Aggregation/Streaming Expressions. Those operate off of docValues
fields anyway, so you kind of get all this "for free".

I don't see the Solr version though, much of this is in later 5x
versions. Pull down the Solr Reference Guide for the version you have
and start there. No matter what, this will be much faster than
paginating.

Best,
Erick

On Thu, Apr 21, 2016 at 6:44 AM, Valentin Popov <va...@gmail.com> wrote:
> Chris , hello.
>
> Thank got the tip, but could you explain how can I use it?
>
> Regards,
> Valentin.
>> On 16 нояб. 2015 г., at 0:42, Chris Hostetter <ho...@fucit.org> wrote:
>>
>>
>> :                     public void collect(int docID) throws IOException {
>> :                             Document doc = indexSearcher.doc(docID, loadFields);
>> :                             found.found(doc);
>> :                     }
>>
>> Based on your description of the calculation you are doing on all of these
>> docs, you will probably find using DocValues on the "to" field and using
>> that in your calculations will be a lot faster then dealing with the
>> StoredFields...
>>
>> : >>>>>> We have ~10 indexes for 500M documents, each document
>> : >>>>>> has «archive date», and «to» address, one of our task is
>> : >>>>>> calculate statistics of «to» for last year. Right now we are
>> : >>>>>> using search archive_date:(current_date - 1 year) and paginate
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Regards,
> Valentin Popov
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org