DocumentCache Out Of Memory

[da...@correo.aeat.es]

Hello,

in our project we need to execute some big queries against Solr once a 
day, with maybe more than 1000 results,  in order to trigger a batch 
proccess with the results. In the fl parameter we only are putting the ID 
field, because we don't need large text fields.

This is our scenary:

- Our documents are generally very big, but as I have said we only request 
for the ID field.
- We have the enableLazyFieldLoading parameter set to true in 
solrconfig.xml, so the DocumentCache should load only the ID field that we 
are requesting.
- Our DocumetCache is set to 8192 objects.
- This test have been executed in Solr 4.2.1, 4.6.1 and 4.7, in no Solr 
Cloud as well as in Solr Cloud mode.


The issue we have got is this:
 
- When we request for more than 1000 docs. more or less JVM takes a lot of 
memory and ends with an OOM.
- Seeing in "real time" as the DocumentCache inserts documents, we have 
seen that memory grows when the documents are bigger (and time needed to 
load that documents in Cache is larger also), but we don't understand why, 
because with enableLazyFieldLoading only the ID should be loaded, so, ¿why 
does memory grow in that way?

We know that one solution is increase RAM memory and other is decrease the 
size of DocumentCache (we have already done this), but we'd like to know 
why this issue with memory.

On the other hand, one good solution for us would be make the queries 
without cache. ¿Is there any way to say Solr not to cache some specific 
queries? I don't think so, but maybe I am wrong.

Thank very much,



David Dávila Atienza
AEAT - Departamento de Informática Tributaria
Subdirección de Tecnologías de Análisis de la Información e Investigación 
del Fraude
Teléfono: 917681160
Extensión: 30160