You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Nordstrom, Karl" <kx...@psu.edu> on 2022/02/07 20:57:07 UTC
GC overhead limit exceeded
Hello activemq users,
We are running v5.16.3 on our amq dev nodes and hit these errors today. Has anybody else encounter these? The heap size on dev is 1G while it's 4G on acceptance. I wonder if 5.16.3 requires more space than 5.15.12 did?
2022-02-07 15:22:39,325 | ERROR | JobScheduler: JMS Caught exception in mainloop | org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.OutOfMemoryError: GC overhead limit exceeded
2022-02-07 15:22:45,761 | ERROR | Checkpoint failed | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
java.lang.OutOfMemoryError: GC overhead limit exceeded
---
Karl Nordström
Systems Administrator
Penn State IT | Application Platforms
Re: GC overhead limit exceeded
Posted by "Nordstrom, Karl" <kx...@psu.edu>.
Thanks Matt, I'll look for GC tuning params for java-1.8.0-openjdk. These hosts do have two processors and 8G of ram.
[kxn2@amq-d02 ~]$ grep processor /proc/cpuinfo
processor : 0
processor : 1
[kxn2@amq-a02 ~]$ grep processor /proc/cpuinfo
processor : 0
processor : 1
[kxn2@amq-p02 ~]$ grep processor /proc/cpuinfo
processor : 0
processor : 1
---
Karl Nordström
Systems Administrator
Penn State IT | Application Platforms
________________________________
From: Matt Pavlovich <ma...@gmail.com>
Sent: Monday, February 7, 2022 9:47 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: GC overhead limit exceeded
Hello Karl-
Those error messages are usually indicative of a garbage collection (GC) tuning problem at the JVM level. On any busy system running Java, the default GC settings may not force garbage collection frequently enough in order to keep memory resources available.
I suggest you look into garbage collection logging and GC tuning options for your version of the JDK.
Thanks,
Matt
> On Feb 7, 2022, at 2:57 PM, Nordstrom, Karl <kx...@psu.edu> wrote:
>
> Hello activemq users,
>
> We are running v5.16.3 on our amq dev nodes and hit these errors today. Has anybody else encounter these? The heap size on dev is 1G while it's 4G on acceptance. I wonder if 5.16.3 requires more space than 5.15.12 did?
>
> 2022-02-07 15:22:39,325 | ERROR | JobScheduler: JMS Caught exception in mainloop | org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> 2022-02-07 15:22:45,761 | ERROR | Checkpoint failed | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
>
> ---
>
> Karl Nordström
>
> Systems Administrator
>
> Penn State IT | Application Platforms
Re: GC overhead limit exceeded
Posted by "Nordstrom, Karl" <kx...@psu.edu>.
Hello,
Here are the memory and cpu stats.
[kxn2@amq-d02 ~]$ free -h
total used free shared buff/cache available
Mem: 7.6G 745M 267M 422M 6.6G 6.2G
Swap: 0B 0B 0B
[kxn2@amq-a02 ~]$ free -h
total used free shared buff/cache available
Mem: 7.6G 1.0G 2.8G 148M 3.8G 6.1G
Swap: 0B 0B 0B
[kxn2@amq-p02 ~]$ free -h
total used free shared buff/cache available
Mem: 7.6G 964M 2.6G 414M 4.1G 6.0G
Swap: 0B 0B 0B
I increased -Xmx from 1G to 2G, on development to match acceptance and production.
[kxn2@amq-d02 activemq]$ grep ACTIVEMQ_OPTS_MEMORY= bin/env
ACTIVEMQ_OPTS_MEMORY="-Xms64M -Xmx2G"
[kxn2@amq-a02 activemq]$ grep ACTIVEMQ_OPTS_MEMORY= bin/env
ACTIVEMQ_OPTS_MEMORY="-Xms64M -Xmx2G"
[kxn2@amq-p02 activemq]$ grep ACTIVEMQ_OPTS_MEMORY= bin/env
ACTIVEMQ_OPTS_MEMORY="-Xms64M -Xmx2G"
---
Karl Nordström
Systems Administrator
Penn State IT | Application Platforms
________________________________
From: Tim Bain <tb...@alumni.duke.edu>
Sent: Tuesday, February 8, 2022 8:08 AM
To: ActiveMQ Users <us...@activemq.apache.org>
Subject: Re: GC overhead limit exceeded
It's probably also worth considering whether you're giving the JVM (and the
machine) enough RAM for the amount of data you're asking it to hold in
memory.
Tim
On Mon, Feb 7, 2022, 7:47 PM Matt Pavlovich <ma...@gmail.com> wrote:
> Hello Karl-
>
> Those error messages are usually indicative of a garbage collection (GC)
> tuning problem at the JVM level. On any busy system running Java, the
> default GC settings may not force garbage collection frequently enough in
> order to keep memory resources available.
>
> I suggest you look into garbage collection logging and GC tuning options
> for your version of the JDK.
>
> Thanks,
> Matt
>
> > On Feb 7, 2022, at 2:57 PM, Nordstrom, Karl <kx...@psu.edu> wrote:
> >
> > Hello activemq users,
> >
> > We are running v5.16.3 on our amq dev nodes and hit these errors today.
> Has anybody else encounter these? The heap size on dev is 1G while it's 4G
> on acceptance. I wonder if 5.16.3 requires more space than 5.15.12 did?
> >
> > 2022-02-07 15:22:39,325 | ERROR | JobScheduler: JMS Caught exception in
> mainloop | org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl |
> JobScheduler:JMS
> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> >
> > 2022-02-07 15:22:45,761 | ERROR | Checkpoint failed |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> >
> >
> > ---
> >
> > Karl Nordström
> >
> > Systems Administrator
> >
> > Penn State IT | Application Platforms
>
>
Re: GC overhead limit exceeded
Posted by Tim Bain <tb...@alumni.duke.edu>.
It's probably also worth considering whether you're giving the JVM (and the
machine) enough RAM for the amount of data you're asking it to hold in
memory.
Tim
On Mon, Feb 7, 2022, 7:47 PM Matt Pavlovich <ma...@gmail.com> wrote:
> Hello Karl-
>
> Those error messages are usually indicative of a garbage collection (GC)
> tuning problem at the JVM level. On any busy system running Java, the
> default GC settings may not force garbage collection frequently enough in
> order to keep memory resources available.
>
> I suggest you look into garbage collection logging and GC tuning options
> for your version of the JDK.
>
> Thanks,
> Matt
>
> > On Feb 7, 2022, at 2:57 PM, Nordstrom, Karl <kx...@psu.edu> wrote:
> >
> > Hello activemq users,
> >
> > We are running v5.16.3 on our amq dev nodes and hit these errors today.
> Has anybody else encounter these? The heap size on dev is 1G while it's 4G
> on acceptance. I wonder if 5.16.3 requires more space than 5.15.12 did?
> >
> > 2022-02-07 15:22:39,325 | ERROR | JobScheduler: JMS Caught exception in
> mainloop | org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl |
> JobScheduler:JMS
> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> >
> > 2022-02-07 15:22:45,761 | ERROR | Checkpoint failed |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> > java.lang.OutOfMemoryError: GC overhead limit exceeded
> >
> >
> > ---
> >
> > Karl Nordström
> >
> > Systems Administrator
> >
> > Penn State IT | Application Platforms
>
>
Re: GC overhead limit exceeded
Posted by Matt Pavlovich <ma...@gmail.com>.
Hello Karl-
Those error messages are usually indicative of a garbage collection (GC) tuning problem at the JVM level. On any busy system running Java, the default GC settings may not force garbage collection frequently enough in order to keep memory resources available.
I suggest you look into garbage collection logging and GC tuning options for your version of the JDK.
Thanks,
Matt
> On Feb 7, 2022, at 2:57 PM, Nordstrom, Karl <kx...@psu.edu> wrote:
>
> Hello activemq users,
>
> We are running v5.16.3 on our amq dev nodes and hit these errors today. Has anybody else encounter these? The heap size on dev is 1G while it's 4G on acceptance. I wonder if 5.16.3 requires more space than 5.15.12 did?
>
> 2022-02-07 15:22:39,325 | ERROR | JobScheduler: JMS Caught exception in mainloop | org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl | JobScheduler:JMS
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> 2022-02-07 15:22:45,761 | ERROR | Checkpoint failed | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
>
> ---
>
> Karl Nordström
>
> Systems Administrator
>
> Penn State IT | Application Platforms