You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@tomcat.apache.org by "Berneburg, Cris J. - US" <cb...@caci.com.INVALID> on 2021/05/27 19:24:27 UTC

[OT] web app big memory usage?

Hi Folks :-)

One of our web apps is using a "lot" of memory, specifically a big user query. We'd like to find out why.

The Tomcat Web Application Manager Find leaks button said that "No web applications appear to have triggered a memory leak on stop, reload or undeploy."

Tomcat Manager Server Status shows that 1.7GB (82%) of G1 Old Gen space is being used that has not been recycled yet.

I grabbed a heap dump and used Eclipse Memory Analyzer, and it shows that only 94MB of memory is being used when G1 Old Gen space used 1.8GB. MAT seems to be looking only at the active objects, not the discarded ones. IOW, we're looking at what the app is doing ATM, not what it already did.

I want to explore the 1.7GB garbage pile to see what's being thrown away, not what things are still being used, to determine wastefulness.

1. Is there a way to analyze uncollected garbage?

2. Is that a reasonable way to identify potential memory usage problems?

Some technical specifics:
* TC 8.5.63
* Java 1.8.0_291
* AWS EC2 instance.
* Windows Server 2016.
* Instance started as Windows Service.
* There are other TC instances on the same server.
* Each TC instance has multiple apps.

Thanks for reading this far. :-)

--
Cris Berneburg
CACI Senior Software Engineer

________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: [OT] web app big memory usage?

Posted by "Berneburg, Cris J. - US" <cb...@caci.com.INVALID>.

Hi John :-)

cb> 1. Is there a way to analyze uncollected garbage?
cb> 2. Is that a reasonable way to identify potential memory usage problems?

jeg> MAT has an option  to "Keep unreachable options."  It's under preferences.

Thanks for the suggestion!  I did not know about that option.

jeg> It sounds like you don't have an actual leak, just high allocation/GC.

Yeah, that's what I think too.

jeg> My favorite tool for this is to use the Java Flight Recorder and analyze it with Java Mission Control.

Hmm... Sounds interesting.  I'll check it out!

--
Cris Berneburg
CACI Senior Software Engineer


________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: [OT] web app big memory usage?

Posted by Jo...@wellsfargo.com.INVALID.

Cris,


> -----Original Message-----
> From: Berneburg, Cris J. - US <cb...@caci.com.INVALID>
> Sent: Thursday, May 27, 2021 2:24 PM
> To: users@tomcat.apache.org
> Subject: [OT] web app big memory usage?
> 
> Hi Folks  :-)
> 
> One of our web apps is using a "lot" of memory, specifically a big user query.
> We'd like to find out why.
> 
> The Tomcat Web Application Manager Find leaks button said that "No web
> applications appear to have triggered a memory leak on stop, reload or
> undeploy."
> 
> Tomcat Manager Server Status shows that 1.7GB (82%) of G1 Old Gen space
> is being used that has not been recycled yet.
> 
> I grabbed a heap dump and used Eclipse Memory Analyzer, and it shows that
> only 94MB of memory is being used when G1 Old Gen space used 1.8GB.
> MAT seems to be looking only at the active objects, not the discarded ones.
> IOW, we're looking at what the app is doing ATM, not what it already did.
> 
> I want to explore the 1.7GB garbage pile to see what's being thrown away,
> not what things are still being used, to determine wastefulness.
> 
> 1. Is there a way to analyze uncollected garbage?
> 
> 2. Is that a reasonable way to identify potential memory usage problems?
> 
> Some technical specifics:
> * TC 8.5.63
> * Java 1.8.0_291
> * AWS EC2 instance.
> * Windows Server 2016.
> * Instance started as Windows Service.
> * There are other TC instances on the same server.
> * Each TC instance has multiple apps.
> 
> Thanks for reading this far.  :-)
> 
> --
> Cris Berneburg
> CACI Senior Software Engineer
> 
> 
> ________________________________
> 
> This electronic message contains information from CACI International Inc or
> subsidiary companies, which may be company sensitive, proprietary,
> privileged or otherwise protected from disclosure. The information is
> intended to be used solely by the recipient(s) named above. If you are not
> an intended recipient, be aware that any review, disclosure, copying,
> distribution or use of this transmission or its contents is prohibited. If you
> have received this transmission in error, please notify the sender
> immediately.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org

MAT has an option  to "Keep unreachable options."  It's under preferences.

It sounds like you don't have an actual leak, just high allocation/GC.  My favorite tool for this is to use the Java Flight Recorder and analyze it with Java Mission Control.

John


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [OT] web app big memory usage?

Posted by Christopher Schultz <ch...@christopherschultz.net>.

Cris,

On 6/3/21 12:25, Berneburg, Cris J. - US wrote:
> cb> StringBuilder - 264MB for the supporting byte array and 264MB for the
> cb> returned String, about 790MB total for that piece of the pie.
> cb> Contents were simply the JSON query results returned to the client.
> cb> No mystery there.
> 
> Also, I noticed that the SB internal memory usage is about 2x the
> size of the actual contents.  Is that because each char is stored as
> 2 bytes for Unicode?  (Not the char array to string conversion, which
> is different.)
You'd have to look more closely at the circumstances. I was about to say 
that SB grows 2x each time is grows, but it doesn't.

> CS> Yep: runaway string concatenation. This is a devolution of the
> CS> "Connector/J reads the whole result set into memory before
> CS> returning" thing I mentioned above. Most JSON endpoints
> CS> return arbitrarily large JSON responses and most client
> CS> applications just go "duh, read the JSON, then process it".
> CS> If your JSON is big, well, then you need a lot of memory to
> CS> store it all if that' who you do things.
> 
> Looking at the contents of the JSON, it's not normalized - a lot of
> redundant metadata.  Hand-editing the JSON for analysis reduced it from
> 135 MB to 26 MB.  Maybe the code that generates it can be improved.

Wow, that's quite an improvement.

> CS> If you want to deal with JSON at scale, you need to process
> CS> it in a streaming fashion. The only library I know that can do
> CS> streaming JSON is Noggit, which was developed for use with
> CS> Solr (I think, maybe it came from elsewhere before that).
> CS> Anyway, it's ... not for the faint of heart. But if you can figure
> CS> out out, you can handle petabytes of JSON with a tiny heap.
> 
> I don't think we need to serve up that much data, but I'm guessing
> we  can do better with what we do serve.  Interesting nonetheless.
> 
> CS> You might want to throttle/serialize queries you expect to
> CS> have big responses so that only e.g. 2 of them can be running
> CS> at a time. Maybe all is well when they come one-at-a-time,
> CS> but if you try to handle 5 concurrent "big responses" you bust
> CS> your heap.
> 
> Hmm... I had not thought of throttling that way, restricting the
> number of concurrent queries.  I was thinking about restricting the
> number of records returned.  Not sure how to handle lots of users
> connected but only a few able to query concurrently.  Different DB
> connection pool with fewer connections for queries?

I think it will be easier for you to restrict the total number of 
connections via your pool than to change your application to e.g. page 
the data, or request smaller chunks, or whatever.

As long as the queries don't take a long time to execute and, once they 
are processed, the data are quickly discarded, I think you'll stabalize 
your application. I think users would prefer slow and reliable over fast 
and sometimes unavailable due to OOME :)

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: [OT] web app big memory usage?

Posted by "Berneburg, Cris J. - US" <cb...@caci.com.INVALID>.

Thanks Chris

[snip, snip, snippety-snip]

CS> What's the database? And the driver?

Oracle 19, oracle.jdbc.OracleDriver - jdbc:oracle:thin.

CS> MySQL Connector/J used to (still does?) read 100% of the results
CS> into the heap before Statement.executeQuery() returns unless you
CS> specifically tell it not to. So if your query returns 1M rows, you
CS> might bust your heap.
CS> It's entirely possible that other drivers do similar things.

The JSON has all the rows, so it appears no pagination is being used on the DB level.

cb> Multiple TC instances
cb> (3) because multiple copies of the apps don't play nice with each
cb> other. That is, we can't just rename the WAR files and expect the
cb> deployed apps to stay inside that context name (I think).

CS> You might want to look into that, eventually. If they aren't playing
CS> together nicely, they are not "good" servlet citizens. Solving those
CS> issues may improve other things. *shrug*

Yeah, I was working on that previously, but attention spans are short, and I got pulled off that task onto - SQUIRREL!

cb> StringBuilder - 264MB for the supporting byte array and 264MB for the
cb> returned String, about 790MB total for that piece of the pie.
cb> Contents were simply the JSON query results returned to the client.
cb> No mystery there.

Also, I noticed that the SB internal memory usage is about 2x the size of the actual contents. Is that because each char is stored as 2 bytes for Unicode? (Not the char array to string conversion, which is different.)

CS> Yep: runaway string concatenation. This is a devolution of the
CS> "Connector/J reads the whole result set into memory before
CS> returning" thing I mentioned above. Most JSON endpoints
CS> return arbitrarily large JSON responses and most client
CS> applications just go "duh, read the JSON, then process it".
CS> If your JSON is big, well, then you need a lot of memory to
CS> store it all if that' who you do things.

Looking at the contents of the JSON, it's not normalized - a lot of redundant metadata. Hand-editing the JSON for analysis reduced it from 135 MB to 26 MB. Maybe the code that generates it can be improved.

CS> If you want to deal with JSON at scale, you need to process
CS> it in a streaming fashion. The only library I know that can do
CS> streaming JSON is Noggit, which was developed for use with
CS> Solr (I think, maybe it came from elsewhere before that).
CS> Anyway, it's ... not for the faint of heart. But if you can figure
CS> out out, you can handle petabytes of JSON with a tiny heap.

I don't think we need to serve up that much data, but I'm guessing we can do better with what we do serve. Interesting nonetheless.

CS> You might want to throttle/serialize queries you expect to
CS> have big responses so that only e.g. 2 of them can be running
CS> at a time. Maybe all is well when they come one-at-a-time,
CS> but if you try to handle 5 concurrent "big responses" you bust
CS> your heap.

Hmm... I had not thought of throttling that way, restricting the number of concurrent queries. I was thinking about restricting the number of records returned. Not sure how to handle lots of users connected but only a few able to query concurrently. Different DB connection pool with fewer connections for queries?

cb> (At least StringBuilder is being
cb> used instead of plus-sign String concatenation.)

CS> In Java "..." + "..." uses a StringBuilder

I did not know that. Or I forgot, in which case I can't tell the diff. :-P

CS> In some code, "..." + "..." is just fine

Often it's run-on sentences of plus-sign concatenation with nested quotes, almost unreadable and even worse for editing. I like to replace with SB for readability and maintainability.

CS> I hate it when someone replaces it with:
CS> String foo = new StringBuilder("bar").append("baz").toString();
CS> because the compiler does the _exact same thing_ and you've
CS> just made the code more difficult to read.

Ahhh, the classic train wreck. :-)

CS> in a *loop*, then replacing it with a StringBuilder is pretty
CS> important for performance, otherwise the compiler will
CS> do something stupid

I believe the technical term for that is "stoopid". :-) Yeah, I like to be strategic about SB's and loops.

CS> You might actually have to start reading some code (shiver!).

"You're ... mocking me." :-) Actually, I might be able to pass it off onto the guy who wrote the library. *phew*

--
Cris Berneburg
CACI Senior Software Engineer

________________________________

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [OT] web app big memory usage?

Posted by Christopher Schultz <ch...@christopherschultz.net>.

Cris,

On 6/1/21 09:17, Berneburg, Cris J. - US wrote:
> Hi Chris
> 
> [lots of snippage]
> 
> cb> One of our web apps is using a "lot" of memory, specifically a big
> cb> user query.  We'd like to find out why.
> cb> 1. Is there a way to analyze uncollected garbage?
> cb> * AWS EC2 instance.
> cb> * There are other TC instances on the same server.
> cb> * Each TC instance has multiple apps.
> 
> cs> What's the goal? Do you just Want To Know, or are you trying
> cs> to solve an actual problem.
> 
> a. Barely enough memory to distribute among the multiple TC instances
> and the apps they support.  A big enough user query (no throttling)
> causes OOME's.  Attempting to determine if the code is being wasteful
> in some way and therefore could be made more efficient.
What's the database? And the driver?

MySQL Connector/J used to (still does?) read 100% of the results into 
the heap before Statement.executeQuery() returns unless you specifically 
tell it not to. So if your query returns 1M rows, you might bust your heap.

It's entirely possible that other drivers do similar things.

> b. It's a dev app server (EC2) which hosts diff stages in the dev
> process - dev, test, and prototype streams.  Multiple TC instances
> (3) because multiple copies of the apps don't play nice with each
> other.  That is, we can't just rename the WAR files and expect the
> deployed apps to stay inside that context name (I think).
You might want to look into that, eventually. If they aren't playing 
together nicely, they are not "good" servlet citizens. Solving those 
issues may improve other things. *shrug*

> c. I don't want to debug the code.  I'm relatively new to the 
> project, unfamiliar with some of the code, and anticipate getting
> lost in the weeds.  See point #1 below.  ;-) >
> cs> If you have a bunch of garbage that's not being cleaned up,
> cs> usually it's because there is simply no need to do so. The GC
> cs> is behaving according to the 3 laws of rob..., er, 3 virtues of
> cs> computing[1]:
> cs>
> cs>    1. Laziness: nothing needs that memory so... meh
> cs>    2. Impatience: gotta clean that Eden space quick
> cs>    3. Hubris: if I ever need more memory, I know where to find it
> 
> cs> [1] http://threevirtues.com/
> 
> Ha ha ha!  :-)
> 
> cs> How long does the query take to run?
> 
> Dunno about the time on the DB query itself.  From the user's point of view, a full minute plus.
> 
> cs> What kind of query is it? Are we talking about something like SQL
> 
> Yup.  Classic RDMS back-end.
> 
> cs> or some in-memory database or something which really does
> cs> take a lot of memory for the application to fulfill the request?
> 
> Nah, nothing that fancy.  The only fancy part is using node.js for the front-end.
> 
> I followed Amit's and John's suggestion of using Eclipse Memory
> Analyzer Tool's "Keep unreachable options" when running a query from
> the app client.  Digging deeper into the Leak Suspects Report, I saw
> a StringBuilder - 264MB for the supporting byte array and 264MB for
> the returned String, about 790MB total for that piece of the pie.
> Contents were simply the JSON query results returned to the client.
> No mystery there.
Yep: runaway string concatenation. This is a devolution of the 
"Connector/J reads the whole result set into memory before returning" 
thing I mentioned above. Most JSON endpoints return arbitrarily large 
JSON responses and most client applications just go "duh, read the JSON, 
then process it".

If your JSON is big, well, then you need a lot of memory to store it all 
if that' who you do things.

If you want to deal with JSON at scale, you need to process it in a 
streaming fashion. The only library I know that can do streaming JSON is 
Noggit, which was developed for use with Solr (I think, maybe it came 
from elsewhere before that). Anyway, it's ... not for the faint of 
heart. But if you can figure out out, you can handle petabytes of JSON 
with a tiny heap.

> I suspect that repeating the process with multiple queries will
> reveal multiple StringBuilder's each containing big honking JSON
> results.

Very likely. You might want to throttle/serialize queries you expect to 
have big responses so that only e.g. 2 of them can be running at a time. 
Maybe all is well when they come one-at-a-time, but if you try to handle 
5 concurrent "big responses" you bust your heap.

The short-term solution is to simply not allow that much concurrency.

> So the issue may not be a problem with efficiency so much as one of
> simple memory hogging.
This can happen with almost any application. We serve ... many clients 
simultaneously and we run with a maximum of 10 connections in our db 
connection pool on each app server. It sounds "too small" but it handles 
the user-traffic without any user complaints.

> (At least StringBuilder is being
> used instead of plus-sign String concatenation.)
In Java "..." + "..." uses a StringBuilder (unless the compiler can 
perform the concatenation at compile-time, in which case there is no 
concatenation at all). In some code, "..." + "..." is just fine and I 
hate it when someone replaces it with:

   String foo = new StringBuilder("bar").append("baz").toString();

because the compiler does the _exact same thing_ and you've just made 
the code more difficult to read.

But if that were in a *loop*, then replacing it with a StringBuilder is 
pretty important for performance, otherwise the compiler will do 
something stupid like this:

String foo = "";
for(String s : [ "a", "b", "c", "d" ... ] ) {
   StringBuilder temp = new StringBuilder(foo);
   temp.append(s);
   foo = temp.toString();
}

Anyhow, the use of StringBuilder is essentially a requirement so don't 
read too much into it being in your heap. You might actually have to 
start reading some code (shiver!).

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: [OT] web app big memory usage?

Posted by "Berneburg, Cris J. - US" <cb...@caci.com.INVALID>.

Hi Chris

[lots of snippage]

cb> One of our web apps is using a "lot" of memory, specifically a big
cb> user query. We'd like to find out why.
cb> 1. Is there a way to analyze uncollected garbage?
cb> * AWS EC2 instance.
cb> * There are other TC instances on the same server.
cb> * Each TC instance has multiple apps.

cs> What's the goal? Do you just Want To Know, or are you trying
cs> to solve an actual problem.

a. Barely enough memory to distribute among the multiple TC instances and the apps they support. A big enough user query (no throttling) causes OOME's. Attempting to determine if the code is being wasteful in some way and therefore could be made more efficient.

b. It's a dev app server (EC2) which hosts diff stages in the dev process - dev, test, and prototype streams. Multiple TC instances (3) because multiple copies of the apps don't play nice with each other. That is, we can't just rename the WAR files and expect the deployed apps to stay inside that context name (I think).

c. I don't want to debug the code. I'm relatively new to the project, unfamiliar with some of the code, and anticipate getting lost in the weeds. See point #1 below. ;-)

cs> If you have a bunch of garbage that's not being cleaned up,
cs> usually it's because there is simply no need to do so. The GC
cs> is behaving according to the 3 laws of rob..., er, 3 virtues of
cs> computing[1]:
cs>
cs> 1. Laziness: nothing needs that memory so... meh
cs> 2. Impatience: gotta clean that Eden space quick
cs> 3. Hubris: if I ever need more memory, I know where to find it

cs> [1] http://threevirtues.com/

Ha ha ha! :-)

cs> How long does the query take to run?

Dunno about the time on the DB query itself. From the user's point of view, a full minute plus.

cs> What kind of query is it? Are we talking about something like SQL

Yup. Classic RDMS back-end.

cs> or some in-memory database or something which really does
cs> take a lot of memory for the application to fulfill the request?

Nah, nothing that fancy. The only fancy part is using node.js for the front-end.

I followed Amit's and John's suggestion of using Eclipse Memory Analyzer Tool's "Keep unreachable options" when running a query from the app client. Digging deeper into the Leak Suspects Report, I saw a StringBuilder - 264MB for the supporting byte array and 264MB for the returned String, about 790MB total for that piece of the pie. Contents were simply the JSON query results returned to the client. No mystery there.

I suspect that repeating the process with multiple queries will reveal multiple StringBuilder's each containing big honking JSON results. So the issue may not be a problem with efficiency so much as one of simple memory hogging. (At least StringBuilder is being used instead of plus-sign String concatenation.)

--
Cris Berneburg
CACI Senior Software Engineer

________________________________

Re: [OT] web app big memory usage?

Posted by Christopher Schultz <ch...@christopherschultz.net>.

Cris,

On 5/27/21 15:24, Berneburg, Cris J. - US wrote:
> Hi Folks  :-)
> 
> One of our web apps is using a "lot" of memory, specifically a big
> user query.  We'd like to find out why.
> 
> The Tomcat Web Application Manager Find leaks button said that "No
> web applications appear to have triggered a memory leak on stop,
> reload or undeploy."
> 
> Tomcat Manager Server Status shows that 1.7GB (82%) of G1 Old Gen
> space is being used that has not been recycled yet.
> 
> I grabbed a heap dump and used Eclipse Memory Analyzer, and it shows
> that only 94MB of memory is being used when G1 Old Gen space used
> 1.8GB.  MAT seems to be looking only at the active objects, not the
> discarded ones.  IOW, we're looking at what the app is doing ATM, not
> what it already did.
> 
> I want to explore the 1.7GB garbage pile to see what's being thrown
> away, not what things are still being used, to determine
> wastefulness.
> 
> 1. Is there a way to analyze uncollected garbage?
> 
> 2. Is that a reasonable way to identify potential memory usage
> problems?
> 
> Some technical specifics: * TC 8.5.63 * Java 1.8.0_291 * AWS EC2
> instance. * Windows Server 2016. * Instance started as Windows
> Service. * There are other TC instances on the same server. * Each TC
> instance has multiple apps.
> 
> Thanks for reading this far.  :-)

What's the goal? Do you just Want To Know, or are you trying to solve an
actual problem.

If you have a bunch of garbage that's not being cleaned up, usually it's
because there is simply no need to do so. The GC is behaving according
to the 3 laws of rob..., er, 3 virtues of computing[1]:

   1. Laziness: nothing needs that memory so... meh
   2. Impatience: gotta clean that Eden space quick
   3. Hubris: if I ever need more memory, I know where to find it

Seriously, though, the only real difference between the new gen and the 
old gen is time, so if your query ran faster it might get a lot more of 
its scratch objects cleaner-up without having to ask for a full GC.

How long does the query take to run? What kind of query is it? Are we 
talking about something like SQL or some in-memory database or something 
which really does take a lot of memory for the application to fulfill 
the request?

[1] http://threevirtues.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: [OT] web app big memory usage?

Posted by "Berneburg, Cris J. - US" <cb...@caci.com.INVALID>.

Hi Amit  :-)

cb> 1. Is there a way to analyze uncollected garbage?
cb> 2. Is that a reasonable way to identify potential memory usage problems?

ap> Have you enabled the " Enable 'keep unreachable objects'" setting of MAT?
ap> https://blog.gceasy.io/2015/12/11/eclipse-mat-titbits/

No, I had not heard of that before.  Thanks for the suggestion!

--
Cris Berneburg
CACI Senior Software Engineer


________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: [OT] web app big memory usage?

Posted by Amit Pande <Am...@veritas.com>.

Have you enabled the " Enable 'keep unreachable objects'" setting of MAT?

https://blog.gceasy.io/2015/12/11/eclipse-mat-titbits/


Thanks,
Amit

-----Original Message-----
From: Berneburg, Cris J. - US <cb...@caci.com.INVALID> 
Sent: Thursday, May 27, 2021 2:24 PM
To: users@tomcat.apache.org
Subject: [EXTERNAL] [OT] web app big memory usage?

Hi Folks  :-)

One of our web apps is using a "lot" of memory, specifically a big user query.  We'd like to find out why.

The Tomcat Web Application Manager Find leaks button said that "No web applications appear to have triggered a memory leak on stop, reload or undeploy."

Tomcat Manager Server Status shows that 1.7GB (82%) of G1 Old Gen space is being used that has not been recycled yet.

I grabbed a heap dump and used Eclipse Memory Analyzer, and it shows that only 94MB of memory is being used when G1 Old Gen space used 1.8GB.  MAT seems to be looking only at the active objects, not the discarded ones.  IOW, we're looking at what the app is doing ATM, not what it already did.

I want to explore the 1.7GB garbage pile to see what's being thrown away, not what things are still being used, to determine wastefulness.

1. Is there a way to analyze uncollected garbage?

2. Is that a reasonable way to identify potential memory usage problems?

Some technical specifics:
* TC 8.5.63
* Java 1.8.0_291
* AWS EC2 instance.
* Windows Server 2016.
* Instance started as Windows Service.
* There are other TC instances on the same server.
* Each TC instance has multiple apps.

Thanks for reading this far.  :-)

--
Cris Berneburg
CACI Senior Software Engineer


________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: [OT] web app big memory usage?

Posted by "Berneburg, Cris J. - US" <cb...@caci.com.INVALID>.

Hi Raghunath

cb> One of our web apps is using a "lot" of memory,
cb> specifically a big user query.  We'd like to find out why.
cb> 1. Is there a way to analyze uncollected garbage?

rm> You could try using the Oracle utility - "jstat"  - for analyzing
rm> the GC in an active Java process (PID)
rm> The "gcold" option helps us to peep into the Old Generation area
rm> jstat -gcold PID
rm> jstat -gcoldcapacity PID
rm>
rm> https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html

That sounds interesting.  :-)  I'll look into it!

--
Cris Berneburg
CACI Senior Software Engineer


________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: [OT] web app big memory usage?

Posted by "Mysore, Raghunath" <rm...@visa.com.INVALID>.

You could try using the Oracle utility - "jstat"  - for analyzing the GC in an active Java process (PID) 
The "gcold" option helps us to peep into the Old Generation area 
jstat -gcold PID 
jstat -gcoldcapacity PID  

https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.html

-----Original Message-----
From: Berneburg, Cris J. - US <cb...@caci.com.INVALID> 
Sent: Thursday, May 27, 2021 1:24 PM
To: users@tomcat.apache.org
Subject: [OT] web app big memory usage?

Hi Folks  :-)

One of our web apps is using a "lot" of memory, specifically a big user query.  We'd like to find out why.

The Tomcat Web Application Manager Find leaks button said that "No web applications appear to have triggered a memory leak on stop, reload or undeploy."

Tomcat Manager Server Status shows that 1.7GB (82%) of G1 Old Gen space is being used that has not been recycled yet.

I grabbed a heap dump and used Eclipse Memory Analyzer, and it shows that only 94MB of memory is being used when G1 Old Gen space used 1.8GB.  MAT seems to be looking only at the active objects, not the discarded ones.  IOW, we're looking at what the app is doing ATM, not what it already did.

I want to explore the 1.7GB garbage pile to see what's being thrown away, not what things are still being used, to determine wastefulness.

1. Is there a way to analyze uncollected garbage?

2. Is that a reasonable way to identify potential memory usage problems?

Some technical specifics:
* TC 8.5.63
* Java 1.8.0_291
* AWS EC2 instance.
* Windows Server 2016.
* Instance started as Windows Service.
* There are other TC instances on the same server.
* Each TC instance has multiple apps.

Thanks for reading this far.  :-)

--
Cris Berneburg
CACI Senior Software Engineer


________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org