Segment Store and unbounded instances of MappedByteBuffer causing high RAM usage

[Chetan Mehrotra <ch...@gmail.com>]

Hi,

On some systems running on Oak (on Windows) where lots of nodes get
created we are seeing high RAM usage. The Java process memory (shown
directly under task manager. See note on Working Set below) remains
within expected range but system starts paging memory quite a bit and
slows down.

On checking the runtime state following things can be observed
* FileStore has 186 TarFile and each TarFile refers to 256MB
MappedByteBuffer taking around 46 GB of memory
* The JVM process constantly shows 50% CPU usage. Checking further
indictes that AsynchUpdate is running and Lucene indexing is being
performed
* Windows Task Manager shows 7 .66 GB (total 8 GB) overall memory
usage with Java process only showing 1.4GB usage. No other process
account for so high usage
* Checking the Working Set [1] shows 7 GB memory being used by Java
process alone

>From my basic understanding of the Memory Mapped file usage it should
not cause msuch resource crunch and OS would be able to manage the
memory. However probably in Oak case all these TarFiles are getting
accessed frequently which leads to these memory mapped pages held in
physical memory.

Should we put upper cap on number of TarFiles opened?

If any other data needs to be collected to determine the problem cause
further then let me know?


Chetan Mehrotra
[1] http://msdn.microsoft.com/en-us/library/windows/desktop/cc441804(v=vs.85).aspx

Re: Segment Store and unbounded instances of MappedByteBuffer causing high RAM usage

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Tue, Nov 19, 2013 at 11:42 PM, Chetan Mehrotra
<ch...@gmail.com> wrote:
> Thats my thinking as well and hence wanted to discuss this on DL
> first. The system here was being subjected to some DAM related tests
> and quite a few binary files were added to it. The only cause of
> concern was that system become quite slow for user response. The GUI
> was very slow and accessing the Sling server running on Oak was also
> very slow. So we need to control some aspect probably. What aspect
> should be controlled I am not sure.

Right, sounds reasonable. Perhaps we should rate-limit the Lucene
indexer so that it won't end up consuming too much resources. Or it
might be enough to just lower the priority of the indexer thread.

> I would keep an eye on this and if such a problem is again reported to
> me would try to collect more data and try to come up with a test for
> that.

This actually sounds like a good use case for a performance benchmark,
i.e. make sure that normal content access performance isn't (too)
adversely affected by the background indexing work.

BR,

Jukka Zitting

Re: Segment Store and unbounded instances of MappedByteBuffer causing high RAM usage

[Chetan Mehrotra <ch...@gmail.com>]

On Tue, Nov 19, 2013 at 11:29 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Hmm, I wonder if we do have a problem here or if the system is just
> working as designed.

Thats my thinking as well and hence wanted to discuss this on DL
first. The system here was being subjected to some DAM related tests
and quite a few binary files were added to it. The only cause of
concern was that system become quite slow for user response. The GUI
was very slow and accessing the Sling server running on Oak was also
very slow. So we need to control some aspect probably. What aspect
should be controlled I am not sure.

I would keep an eye on this and if such a problem is again reported to
me would try to collect more data and try to come up with a test for
that.

Chetan Mehrotra

Re: Segment Store and unbounded instances of MappedByteBuffer causing high RAM usage

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Tue, Nov 19, 2013 at 1:57 AM, Chetan Mehrotra
<ch...@gmail.com> wrote:
> On some systems running on Oak (on Windows) where lots of nodes get
> created we are seeing high RAM usage. The Java process memory (shown
> directly under task manager. See note on Working Set below) remains
> within expected range but system starts paging memory quite a bit and
> slows down.
>
> On checking the runtime state following things can be observed
> * FileStore has 186 TarFile and each TarFile refers to 256MB
> MappedByteBuffer taking around 46 GB of memory
> * The JVM process constantly shows 50% CPU usage. Checking further
> indictes that AsynchUpdate is running and Lucene indexing is being
> performed
> * Windows Task Manager shows 7 .66 GB (total 8 GB) overall memory
> usage with Java process only showing 1.4GB usage. No other process
> account for so high usage
> * Checking the Working Set [1] shows 7 GB memory being used by Java
> process alone
>
> From my basic understanding of the Memory Mapped file usage it should
> not cause msuch resource crunch and OS would be able to manage the
> memory. However probably in Oak case all these TarFiles are getting
> accessed frequently which leads to these memory mapped pages held in
> physical memory.

Hmm, I wonder if we do have a problem here or if the system is just
working as designed. If you have 8GB of memory on the system and no
other processes using much of it, then it would be natural and
expected for the operating system to use most of the available memory
for caching the memory mapped files that are actively being accessed
by TarMK.

The 50% CPU usage you mention suggests that the system isn't actually
being blocked by the paging behavior. Instead it's just busy crunching
through the content, which causes newly accessed data to be paged in
and old to be discarded. The same disk IO would be required without
memory mapping. If thrashing was occurring, you'd expect to see much
lower CPU usage.

> Should we put upper cap on number of TarFiles opened?

If this indeed is a problem, a better solution would probably be to
limit the maximum working set size at the operating system level.
Alternatively we could switch to using normal file IO for bulk
segments (ones that only contain parts of large binaries).

> If any other data needs to be collected to determine the problem cause
> further then let me know?

Can you reduce the behavior to a standalone test case? Or you might
want to try running the system with a 32 bit JVM, which should force
TarMK to use normal file IO with a fixed-size buffer instead of memory
mapping.

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/ms686237(v=vs.85).aspx

BR,

Jukka Zitting