Re: Regarding maximum number of documents that can be returned safely from SOLR to Java Application.

[Neha Gupta <ne...@uni-jena.de>]

Hello All,

Thanks a lot for the replies.

So the conclusion i came is to use either /exporthandler or cursor for 
fetching large number of documents so i am going to use that and will 
try to run tests to see whether all works fine or not.


Thanks again for all the help.

Regards

Neha Gupta

On 29/04/2022 04:13, David Hastings wrote:
> the 30+ million records I retrieved were always from a single standalone
> solr node, and yes you can do that frequently and it doesnt have an impact
> on the rest of the searches happening assuming you have enough memory to
> deal with it.  there is nothing wrong with requesting every one of your
> documents as well as every single stored field.  It simply does just work.
>
> so like was stated before, just try it and make a run and see what
> happens.
>
> On Thu, Apr 28, 2022 at 7:39 PM Yonik Seeley <ys...@gmail.com> wrote:
>
>> It depends ;-)
>>
>> If you are directly querying a single Solr node, then the additional memory
>> usage is (max_results * 4) if not retrieving scores.  It's just
>> a single int per document to keep track of the docids that matched the
>> query.  Documents are "streamed" to the client... the
>> actual stored fields for each document are only loaded when needed to write
>> to the output stream. If one is retrieving scores as well,
>> then the memory usage is (max_results * 8) (4 bytes for the int id, 4 bytes
>> for the float score.)
>>
>> However, if one is using distributed search, then the entire response *is*
>> aggregated in memory before sending back to the client.
>> So if you are using SolrCloud and wish to do big bulk operations like this,
>> target individual nodes with distrib=false.
>>
>> -Yonik
>>
>>
>> On Wed, Apr 27, 2022 at 4:23 PM Neha Gupta <ne...@uni-jena.de> wrote:
>>
>>> Dear Solr Community,
>>>
>>> I would like to know what is the safe number of documents that can be
>>> returned from a SOLR.
>>>
>>> Just for information I will be firing queries from Java application to
>>> SOLR using SOLRJ and would like to know how much maximum documents (i.e
>>> maximum number of rows that i can request in the query) can be returned
>>> safely from SOLR.
>>>
>>> It would be great if you can please share your experience with regard to
>>> the same.
>>>
>>>
>>> Thanks and Regards
>>> Neha Gupta
>>>
>>>