You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "James H. H. Lampert" <ja...@touchtonecorp.com.INVALID> on 2024/07/23 17:44:02 UTC
Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
Ladies and Gentlemen:
We still have a chronic Tomcat crashing problem at one of our installations.
The weirdest thing about this is that while this is certainly *one* of
our heaviest-usage installations, it's not *the* heaviest.
We already have Tomcat shutting down and restarting itself every night.
And we have the Catalina job and its associated JVM job running in a
private subsystem, with a 7G private memory pool before it starts to dip
into the base memory pool. And we're launching with (according to
catalina.out) -Xms4096m Xmx5120m.
Our webapp has integration with M$ Office 365, which this installation uses.
The usual pattern when it starts to get into trouble may be connected
with that integration. Looking at a typical crash in catalina.out, I see
several OAuth2 errors that appear to involve an expired token, producing
lengthy (over 50 line) Java stacktraces.
Other errors seem to involve messages from graph.microsoft.com involving
"item not found," that seem to be connected with email attachment
downloads from Office365.
Then a NullPointerException is thrown, producing a stacktrace of over 60
lines.
Then another Microsoft "item not found," like the previous one.
Then a handshaking error. Not sure what the handshaking error is *with.*
Then Tomcat runs out of memory in the Java heap space, does a dump, and
everything hits the proverbial fan. 4775 lines of catalina.out entries
before we manually shut it down with extreme prejudice and restart it,
508 of them before the first out-of-memory error, the rest after.
And yes, I've packaged up an excerpt to send to our webapp developers,
to see if they can make head or tail of what went wrong.
Anybody have any suggestions of what to look for?
--
JHHL
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
Posted by "Thomas Hoffmann (Speed4Trade GmbH)" <Th...@speed4trade.com.INVALID>.
Hello James,
> -----Ursprüngliche Nachricht-----
> Von: James H. H. Lampert <ja...@touchtonecorp.com.INVALID>
> Gesendet: Dienstag, 23. Juli 2024 19:44
> An: Midrange Systems Technical Discussion <midrange-
> l@lists.midrange.com>; Tomcat Users List <us...@tomcat.apache.org>
> Betreff: Problems with the most problematic of our Tomcat installations on
> IBM Midrange (cross-posted to Midrange and Tomcat Lists)
>
> Ladies and Gentlemen:
>
> We still have a chronic Tomcat crashing problem at one of our installations.
>
> The weirdest thing about this is that while this is certainly *one* of our
> heaviest-usage installations, it's not *the* heaviest.
>
> We already have Tomcat shutting down and restarting itself every night.
> And we have the Catalina job and its associated JVM job running in a private
> subsystem, with a 7G private memory pool before it starts to dip into the base
> memory pool. And we're launching with (according to
> catalina.out) -Xms4096m Xmx5120m.
>
> Our webapp has integration with M$ Office 365, which this installation uses.
>
> The usual pattern when it starts to get into trouble may be connected with
> that integration. Looking at a typical crash in catalina.out, I see several OAuth2
> errors that appear to involve an expired token, producing lengthy (over 50
> line) Java stacktraces.
>
> Other errors seem to involve messages from graph.microsoft.com involving
> "item not found," that seem to be connected with email attachment
> downloads from Office365.
>
> Then a NullPointerException is thrown, producing a stacktrace of over 60 lines.
>
> Then another Microsoft "item not found," like the previous one.
>
> Then a handshaking error. Not sure what the handshaking error is *with.*
>
> Then Tomcat runs out of memory in the Java heap space, does a dump, and
If you have a dump on disk, this would be a good start for further investigation.
I usually use MAT https://eclipse.dev/mat/ to analyze the dump and figure out what is occupying the memory.
After an OOM, the application is usually not stable anymore because you don’t know beforehand which threads exited.
Therefore, I would focus on the memory issue.
> everything hits the proverbial fan. 4775 lines of catalina.out entries before we
> manually shut it down with extreme prejudice and restart it,
> 508 of them before the first out-of-memory error, the rest after.
>
> And yes, I've packaged up an excerpt to send to our webapp developers, to
> see if they can make head or tail of what went wrong.
>
> Anybody have any suggestions of what to look for?
>
> --
> JHHL
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
Posted by Christopher Schultz <ch...@christopherschultz.net>.
James,
On 7/24/24 18:07, James H. H. Lampert wrote:
>> So what jobs are in the subsystem? You said "the Catalina job and its
>> associated JVM job" but to me those are just a single job/process. Are
>> they separate things in the IBM world?
>
> Thanks for your insights, Mr. Schultz. And yours, too, Herr Hoffmann.
>
> On an IBM Midrange box (AS/400, iSeries, whatever IBM is calling it this
> week), when a shell script (a non-native construct in the IBM Midrange
> world - the native equivalent is something called a CL program, which is
> a compiled object) spawns off a JVM process, it shows up as a separate
> job, typically with the rather cryptic name QP0ZSPWT.
>
> As I recall, when we first started using Tomcat (6, I think!) on IBM
> Midrange boxes, the Catalina job would terminate after spawning off the
> JVM job. Or if you called catalina.sh from a terminal session, you'd get
> your command line back. This changed with (I think) Tomcat 7.
>
> At any rate, that's literally all that's in the dedicated subsystem
> (which is fairly easy to set up on an IBM Midrange box; just create a
> subsystem description, with a private memory pool, if desired, give it a
> routing entry and a job queue, do a STRSBS on it, and you've got your
> subsystem, along with a way to send jobs into it).
>
> At any rate, we don't currently have any dumps on that particular
> customer box, because unless we're planning on having dumps analyzed,
> they're just wasting disk on a box that is not exactly hip-deep in
> unused mass storage. Hopefully, the next time this happens, it will
> happen at a time when our webapp developers aren't hip-deep in other
> projects (famous last words).
If you have access to the box, you can trigger a heap dump yourself
anytime you want. You will need enough disk space for the dump, so if
you have a 5ish GiB heap, be prepared for a big ole' file.
Then you can present it to your dev team when they are ready :)
You can also configure the JVM to dump the heap on OOME which it sounds
like you already have. It you have one or two of them laying around, you
should look at them. :)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
Posted by "James H. H. Lampert" <ja...@touchtonecorp.com.INVALID>.
> So what jobs are in the subsystem? You said "the Catalina job and its
> associated JVM job" but to me those are just a single job/process. Are
> they separate things in the IBM world?
Thanks for your insights, Mr. Schultz. And yours, too, Herr Hoffmann.
On an IBM Midrange box (AS/400, iSeries, whatever IBM is calling it this
week), when a shell script (a non-native construct in the IBM Midrange
world - the native equivalent is something called a CL program, which is
a compiled object) spawns off a JVM process, it shows up as a separate
job, typically with the rather cryptic name QP0ZSPWT.
As I recall, when we first started using Tomcat (6, I think!) on IBM
Midrange boxes, the Catalina job would terminate after spawning off the
JVM job. Or if you called catalina.sh from a terminal session, you'd get
your command line back. This changed with (I think) Tomcat 7.
At any rate, that's literally all that's in the dedicated subsystem
(which is fairly easy to set up on an IBM Midrange box; just create a
subsystem description, with a private memory pool, if desired, give it a
routing entry and a job queue, do a STRSBS on it, and you've got your
subsystem, along with a way to send jobs into it).
At any rate, we don't currently have any dumps on that particular
customer box, because unless we're planning on having dumps analyzed,
they're just wasting disk on a box that is not exactly hip-deep in
unused mass storage. Hopefully, the next time this happens, it will
happen at a time when our webapp developers aren't hip-deep in other
projects (famous last words).
--
JHHL
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
Posted by Christopher Schultz <ch...@christopherschultz.net>.
James,
On 7/23/24 16:54, James H. H. Lampert wrote:
> On 7/23/24 1:25 PM, Christopher Schultz wrote:
>> Thomas,
>
> Uh, "James." Thomas was someone who answered earlier.
:|
>> 2. What has to fit into that 7GiB private memory pool? Does it include
>> any OS, or is it just the JVM itself?
>
> On an IBM Midrange box, a private memory pool simply provides jobs in a
> given subsystem with memory that's off-limits to jobs outside that
> subsystem. If the subsystem description's memory pool usage is set up
> correctly, it will grab memory from its private pool whenever possible,
> before going to the public "base" memory pool on the system. This
> reduces the number of page faults.
Okay.
So what jobs are in the subsystem? You said "the Catalina job and its
associated JVM job" but to me those are just a single job/process. Are
they separate things in the IBM world?
> And based on the back-end server jobs, there are, at last count, 58
> users currently signed on, each one with a dedicated connection to a
> back-end server job. (The webapp is itself a client to a back-end server
> that was originally designed for a desktop Java client.)
That doesn't sound like a lot of users. Will it grow and ultimately fail
with zero users? How about just one? If you really load it down, will it
fail faster?
Do you have any kind of memory monitoring of the JVM process going on,
or do you just have the heap-dump-on-oome configured?
The heap dump will tell your developers A LOT. Any decent heap-dump
analyzer will be able to tell you what's taking up all the memory pretty
quickly. Any time I've ever helped anyone look at a Java heap problem,
one of the following checks always points directly to the problem:
1. One object (and tree of references) that is particularly large.
or
2. One kind of object for which there are a HUGE number of instances.
Sometimes there is a lot of overlap between these two.
When you load the heap dump, you'll notice that there are a huge number
of char[], byte[], and String instances. Don't panic (yet). That's
totally normal.
Even VisualVM can read heap dumps and give you some good information.
Try looking for "Dominators".
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
Posted by "James H. H. Lampert" <ja...@touchtonecorp.com.INVALID>.
On 7/23/24 1:25 PM, Christopher Schultz wrote:
> Thomas,
Uh, "James." Thomas was someone who answered earlier.
> 2. What has to fit into that 7GiB private memory pool? Does it include
> any OS, or is it just the JVM itself?
On an IBM Midrange box, a private memory pool simply provides jobs in a
given subsystem with memory that's off-limits to jobs outside that
subsystem. If the subsystem description's memory pool usage is set up
correctly, it will grab memory from its private pool whenever possible,
before going to the public "base" memory pool on the system. This
reduces the number of page faults.
And based on the back-end server jobs, there are, at last count, 58
users currently signed on, each one with a dedicated connection to a
back-end server job. (The webapp is itself a client to a back-end server
that was originally designed for a desktop Java client.)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problems with the most problematic of our Tomcat installations on IBM Midrange (cross-posted to Midrange and Tomcat Lists)
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Thomas,
On 7/23/24 13:44, James H. H. Lampert wrote:
> Ladies and Gentlemen:
>
> We still have a chronic Tomcat crashing problem at one of our
> installations.
>
> The weirdest thing about this is that while this is certainly *one* of
> our heaviest-usage installations, it's not *the* heaviest.
>
> We already have Tomcat shutting down and restarting itself every night.
> And we have the Catalina job and its associated JVM job running in a
> private subsystem, with a 7G private memory pool before it starts to dip
> into the base memory pool. And we're launching with (according to
> catalina.out) -Xms4096m Xmx5120m.
Might I make a few recommendations and comments, here?
1. Don't bother setting the min and max heap sizes different from each
other. Assuming the process is going to grow to the max heap, you're
going to need the max anyway so you may as well allocate it all at once
up front.
2. What has to fit into that 7GiB private memory pool? Does it include
any OS, or is it just the JVM itself?
3. Note that JVM memory requirements aren't limited to the heap. There's
plenty of "native memory" that is necessary to run a JVM as well. For
example, I have a production application with heap settings -Xms2048M
-Xmx2048M and 'ps' tells me that the virtual size of the process is
9.9GiB and the resident size is 2.7GiB. With a ~5GiB heap, you run the
risk of hitting your memory limit. If you are getting "OOME heap" then
this is not your issue, but it's something to think about.
> Our webapp has integration with M$ Office 365, which this installation
> uses.
>
> The usual pattern when it starts to get into trouble may be connected
> with that integration. Looking at a typical crash in catalina.out, I see
> several OAuth2 errors that appear to involve an expired token, producing
> lengthy (over 50 line) Java stacktraces.
>
> Other errors seem to involve messages from graph.microsoft.com involving
> "item not found," that seem to be connected with email attachment
> downloads from Office365.
>
> Then a NullPointerException is thrown, producing a stacktrace of over 60
> lines.
Long stack traces are not uncommon in web-based applications, especially
if an application framework is involved and/or if you have many
"forward" operations where a request gets forwarded through several
components before a response is finally generated.
I wouldn't draw too many conclusions from the stack-trace size(s).
> Then another Microsoft "item not found," like the previous one.
>
> Then a handshaking error. Not sure what the handshaking error is *with.*
>
> Then Tomcat runs out of memory in the Java heap space, does a dump, and
> everything hits the proverbial fan. 4775 lines of catalina.out entries
> before we manually shut it down with extreme prejudice and restart it,
> 508 of them before the first out-of-memory error, the rest after.
>
> And yes, I've packaged up an excerpt to send to our webapp developers,
> to see if they can make head or tail of what went wrong.
>
> Anybody have any suggestions of what to look for?
As Thomas suggested, a heap dump would be helpful but if you have
already provided that to your application developers there is no need to
send it to us. But... if they aren't sure how to read it or aren't sure
what it's telling them... perhaps you should send THEM to *us*.
It's helpful to know some more about the application. How many users do
you typically serve at a time (concurrent logins, not necessarily
concurrent requests)? What kind of information gets stored long-term in
the application's heap? This would be things such as session storage
(per user, should be freed on logout) or caches (typically global,
either never freed or some kind of eviction policy like "last 1000
entries"). Sometimes, the application really is just storing too much in
memory.
I had an application that ran for years very happily with 64MiB of heap.
One day it started throwing OOMEs and when we investigated the cause, it
turns out we just had more users than we had in the past and we simply
needed to resize of heap.
(A much later analysis showed that handing much more heap to the memory
manager made the GC run a lot more efficiently and "stole" less time
away from the application over time. So we run with a 2GiB heap now,
even though we don't need nearly that much operating memory.)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org