Memory footprint of individual indices at runtime

[Florian Buetow <fb...@mimecast.com>]

Hi,

I would like to know (or estimate) how much memory an opened index consumes inside the JVM (heap) and outside the JVM (fs buffers?).

To my understanding the amount of memory inside the JVM depends on performed searches and search results which might be cached by Lucene.
However, I am not yet able to see a way to get that information on a per index basis when multiple indices have been opened and queried.

I would be grateful if someone could point me into the right direction for this one.

Many thanks!

Cheers

Florian

Florian Buetow
Software Engineer
m: +44 7702 557267
p: +44 207 847 8700
w: www.mimecast.com
Address click here: www.mimecast.com/About-us/Contact-us/

Re: Memory footprint of individual indices at runtime

[Adrien Grand <jp...@gmail.com>]

Filesystem cache usage depends on usage indeed. There are no requirements,
but this is definitely an important performance factor: the more you give
to the fs cache, the better. It is hard to figure out how much is used
specifically for your Lucene indices, but if your server does not run any
other database or disk-intensive process then assuming everything is used
for the Lucene indices should be a good approximation.

JVM heap usage does not depend on usage, you can get an estimate using
CodecReader.ramBytesUsed (you will likely need to iterate over leaves using
IndexReader#leaves() and then cast the reader to a CodecReader).

https://lucene.apache.org/core/6_5_1/core/org/apache/lucene/index/CodecReader.html#ramBytesUsed--

Le lun. 5 juin 2017 à 19:09, Florian Buetow <fb...@mimecast.com> a écrit :

> Hi,
>
>
>
> I would like to know (or estimate) how much memory an opened index
> consumes inside the JVM (heap) and outside the JVM (fs buffers?).
>
>
>
> To my understanding the amount of memory inside the JVM depends on
> performed searches and search results which might be cached by Lucene.
>
> However, I am not yet able to see a way to get that information on a per
> index basis when multiple indices have been opened and queried.
>
>
>
> I would be grateful if someone could point me into the right direction for
> this one.
>
>
>
> Many thanks!
>
>
>
> Cheers
>
>
>
> Florian
>
>
> Florian Buetow m: +44 7702 557267 <+44%207702%20557267> www.mimecast.com
> Software Engineer p: +44 207 847 8700 <+44%2020%207847%208700> Address
> click here <http://www.mimecast.com/About-us/Contact-us/>
> ------------------------------
> [image: http://www.mimecast.com]
> <https://eu-api.mimecast.com/s/click/1K7xTdhoqgjnB3PEFCIbOfs6pqJfJ5-KyvuU1EzpzSDZR4omP_p_W3H1qOb3A9F0LyOP0UeodX7wHHocTgWVUgkY8NJrtqzANffE8SNe39fwg2dMwFSMohTAbio0aJlWWXnbk_HLofQ0NceKQst6WD8wnbN3rdIPU7yCPjN4mL-QBX8bxWh6XlyWw3HRVF-A>
>
> [image: LinkedIn]
> <https://eu-api.mimecast.com/s/click/IKmtIWZgah9PkQKTA72V5yPl_EKlcVKog2WHKhOyU8e6JlDRwF_J2Pqg9zZIuhRSup3zeuQ_LU5IhYmQ_UnVnHbTWqt3RetW99FMW-pRWIsbxwtkv9vtVFcA0ayiJ6c78ScaGmphNyq80dAcyEM3kZVsNDinQVcVUY81egIrZvqdiQqusmsfHhWHAo4vGmkuUWsThNMfSE_TxYg9ZhC-Fg>
> [image: YouTube]
> <https://eu-api.mimecast.com/s/click/F2A44qlyvx7D1oreXULOBVytfno4F0-y3SzqsGatx94PIdMutk0Eh4SVFqtRbE8HIkk8vkxe6F55pokaird9PbwbkTeK6xzoDkgbEf3Jv7JREGC5hEC5JoZtd0TDhar7QYGLsysA_qxqzLlmgHh0s0QhvGUnBXihs0pinvg0j4C0ZXliwXM4WLKge6vV8tQI_Efap0jtj_F2Y2tPwGQzjg>
> [image: Facebook]
> <https://eu-api.mimecast.com/s/click/1K7xTdhoqgjnB3PEFCIbOcRWxcV7-NK_sNSELxAGABarmiDkrKTJpFb8SM6re-1KExc3WgEeAldj9EFcDgAeWO2s9SH_hYH2iwxvRGoY5mokbFgbAtB0ZBIj1hhD9O-jJmwVr6vodqObVmLmjM1AGQ5NOR2NygUSOQ8KR4z76wR3RynGVSBnlFHyRBxic0pcUWsThNMfSE_TxYg9ZhC-Fg>
> [image: Blog]
> <https://eu-api.mimecast.com/s/click/1K7xTdhoqgjnB3PEFCIbOfg1bbSfCf-CnBLCAQNyqWumFi3ZT0dvIJJzxBZ6NxGBUggU2_6hWzcBNwY15ebRsVugPu-u_o4Ft289V51dvTGrikke8OxYjvBP_lqjRShfjOqSPa8hxLvrVnVQr8W0DdMnuOQ8nBBW2YIHKNoL9HjkwW0OFjRE2ipUmpO6F2LOGXeltlXhMl7zRxl0nGry1A>
> [image: Twitter]
> <https://eu-api.mimecast.com/s/click/ta4Eoos53TWGGPy2c4zTOFDFpH818UEPNFrk2CPr6VCCJjwSiX-H57UE8Z52A0Q8-cudk9-Gk5Whfjolc7v3gduI6VQJFN9VZg0WqD8NMobRo0Q8JAUY2tCbGY_Hr3VuFKPEpCnOqoPiG_IaJl_WEWHAm3R76F0OxQKnt1xpcJXV_SKKxU-R7zZNUn-ZeA2_ogk8xV7WZoS-auXQ1CDvwg>
>
>
>
>