Excessive mem usage with 32-bit app, on 64-bit server

[Antony Joseph <an...@gmail.com>]

Hi,

We have a 32-bit python app which uses pylucence and bundles the JRE with it.

This is installed on a 64-bit Windows 2012 Server with 16GB of RAM.
The heap settings are: -Xms=64Mb and -Xmx=1024Mb

It's a web app and works fine when a user or two connect. But if there
are a few more users, the memory usage increases and at some point
there is an "Out of memory error".

By and large, a single index searcher is used and shared amongst
multiple queries.

While checking the Net someone said that with Lucene there is a lot of
memory usage behind the scene (it needs to load some data structures
into memory, etc.) - the questions that we have now include:

1. Does memory usage go up with multiple simultaneous searches - does
it need to load the data structures multiple times?
2. Has this got anything to do with a 32-bit app on a 64-bit architecture?
3. Normally, after a search query is executed, when does the memory
used by the result get free again? Is it after an idle period or when
the JVM hits memory usage limits or what?
4. Could this be caused due to a memory leak in our code? Any "common
mistakes" that we could check first?

Thanks,

Antony

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Excessive mem usage with 32-bit app, on 64-bit server

[Ian Lea <ia...@gmail.com>]

> 1. Does memory usage go up with multiple simultaneous searches - does
> it need to load the data structures multiple times?

Lucene loads some stuff into RAM, but just once rather than for each
search.  But there will of course be memory used for each search, more
concurrent searches will use more.

> 2. Has this got anything to do with a 32-bit app on a 64-bit architecture?

No idea on that, but if so you might have thought it would be an all
or nothing problem, unrelated to concurrent searches.

> 3. Normally, after a search query is executed, when does the memory
> used by the result get free again? Is it after an idle period or when
> the JVM hits memory usage limits or what?

Up to the JVM garbage collector.

> 4. Could this be caused due to a memory leak in our code? Any "common
> mistakes" that we could check first?

Yes, it certainly could be a problem with your code.  Are you hanging
on to result lists or related structures longer than necessary?
Running a memory profiler would surely help.


There are useful for info on memory usage:
http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html
and http://searchhub.org/2011/09/14/estimating-memory-and-storage-for-lucenesolr/


--
Ian.


On Thu, Nov 22, 2012 at 1:16 PM, Antony Joseph <an...@gmail.com> wrote:
>
> Hi,
>
> We have a 32-bit python app which uses pylucence and bundles the JRE with
> it.
>
> This is installed on a 64-bit Windows 2012 Server with 16GB of RAM.
> The heap settings are: -Xms=64Mb and -Xmx=1024Mb
>
> It's a web app and works fine when a user or two connect. But if there
> are a few more users, the memory usage increases and at some point
> there is an "Out of memory error".
>
> By and large, a single index searcher is used and shared amongst
> multiple queries.
>
> While checking the Net someone said that with Lucene there is a lot of
> memory usage behind the scene (it needs to load some data structures
> into memory, etc.) - the questions that we have now include:
>
> 1. Does memory usage go up with multiple simultaneous searches - does
> it need to load the data structures multiple times?
> 2. Has this got anything to do with a 32-bit app on a 64-bit architecture?
> 3. Normally, after a search query is executed, when does the memory
> used by the result get free again? Is it after an idle period or when
> the JVM hits memory usage limits or what?
> 4. Could this be caused due to a memory leak in our code? Any "common
> mistakes" that we could check first?
>
> Thanks,
>
> Antony
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org