You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ofbiz.apache.org by Jacques Le Roux via dev <de...@ofbiz.apache.org> on 2026/10/17 16:29:14 UTC
OFBiz demos are very slow or down
Hi,
Between 16:45 UTC and "now", OFBiz demos are slow (or even very slow) or down.
Trunk is by far the much affected, which is weird... I'll have a look...
Jacques
Re: OFBiz demos are very slow or down
Posted by Jacques Le Roux via dev <de...@ofbiz.apache.org>.
Le 17/10/2026 à 20:52, Jacques Le Roux via dev a écrit :
> BTW we need to update tools\demo-backup\ofbizdocker\README.md
>
> The VM is no longer ofbiz-vm1.apache.org, it's now ofbiz-ec2-or.apache.org
Oops forget it, I did that already the 2026/4/8
Re: OFBiz demos are very slow or down
Posted by Jacques Le Roux via dev <de...@ofbiz.apache.org>.
Le 17/10/2026 à 21:03, Jacques Le Roux via dev a écrit :
> Le 17/10/2026 à 20:52, Jacques Le Roux via dev a écrit :
>> Also not sure why, I was unable to sudo as ofbizdocker using the same than last Challenge password.
>
> Actually I vaguely remember having changed the passphrase used as Challenge password. Not even sure of that 😳.
>
> Anyway as long as you are caring about the demos I'm quite relax :)
>
> Jacques
Actually I remember, it's simple. I just have to reset the OTP passphrase used to feed the Challenge password.
Re: OFBiz demos are very slow or down
Posted by Jacques Le Roux via dev <de...@ofbiz.apache.org>.
Le 17/10/2026 à 20:52, Jacques Le Roux via dev a écrit :
> Also not sure why, I was unable to sudo as ofbizdocker using the same than last Challenge password.
Actually I vaguely remember having changed the passphrase used as Challenge password. Not even sure of that 😳.
Anyway as long as you are caring about the demos I'm quite relax :)
Jacques
Re: OFBiz demos are very slow or down
Posted by Jacques Le Roux via dev <de...@ofbiz.apache.org>.
Hi Daniel,
Forgot to send my email (was hungry ;), so we crossed on wire. Thanks for looking at it. IMO, it's clearly easier to increase the DB memory limit, as
least for now.
Looking at https://montastic.com/checkpoints I confirm that all is OK now.
BTW we need to update tools\demo-backup\ofbizdocker\README.md
The VM is no longer ofbiz-vm1.apache.org, it's now ofbiz-ec2-or.apache.org
Also not sure why, I was unable to sudo as ofbizdocker using the same than last Challenge password.
Thanks and cheers
Jacques
Le 17/10/2026 à 19:57, Daniel Watford a écrit :
> Summary: Postgres database exceeded its 300 MB memory limit, causing the kernel to kill the process. Upon restart, the database failed to recover
> from corruption.
> Resolved by: Tearing down and restarting demos. This caused the databases to reset.
> To consider: Perhaps the 300 MB limit for the database process is too low? Or perhaps some run-away process in object caused excessive database use?
>
> Hi Jacques,
>
> I took a look at the standard output and standard error logs for ofbiz-trunk and found there was a huge amount, suggesting some sort of cycle.
>
> Filtering the logs for the Postgres database, I used the command:
>
> docker compose logs db | less
>
> Extract from the logs:
> db-1 | 2026-10-17 02:35:56.952 UTC [64] LOG: database system was shut down at 2026-10-17 02:35:56 UTC
> db-1 | 2026-10-17 02:35:56.962 UTC [1] LOG: database system is ready to accept connections
> db-1 | 2026-10-17 11:47:24.253 UTC [1] LOG: server process (PID 1747) was terminated by signal 9: Killed
> db-1 | 2026-10-17 11:47:24.253 UTC [1] DETAIL: Failed process was running: SELECT PTYACCPREF.PARTY_ID, PTYACCPREF.BASE_CURRENCY_UOM_ID,
> PTYGROUP.GROUP_NAME, PTYROLE.ROLE_TYPE_ID FROM (public.PARTY_ACCTG_PREFERENCE PTYACCPREF INNER JOIN public.PARTY_GROUP PTYGROUP ON
> PTYACCPREF.PARTY_ID = PTYGROUP.PARTY_ID) INNER JOIN public.PARTY_ROLE PTYROLE ON PTYACCPREF.PARTY_ID = PTYROLE.PARTY_ID WHERE
> (((PTYROLE.ROLE_TYPE_ID = $1)))
> db-1 | 2026-10-17 11:47:24.253 UTC [1] LOG: terminating any other active server processes
> db-1 | 2026-10-17 11:47:24.253 UTC [2369] WARNING: terminating connection because of crash of another server process at character 245
> db-1 | 2026-10-17 11:47:24.253 UTC [2369] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit,
> because another server process exited abnormally and possibly corrupted shared memory.
> db-1 | 2026-10-17 11:47:24.253 UTC [2369] HINT: In a moment you should be able to reconnect to the database and repeat your command.
> db-1 | 2026-10-17 11:47:24.253 UTC [2357] WARNING: terminating connection because of crash of another server process
> db-1 | 2026-10-17 11:47:24.253 UTC [2357] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit,
> because another server process exited abnormally and possibly corrupted shared memory.
>
> It appears that the database was killed at 11:47:24.253 UTC. The question is why....
>
> Searching in /var/log/syslog for 2026-10-17T11:47 we find:
>
> 2026-10-17T11:47:24.235788+00:00 ofbiz-ec2-or kernel: postgres invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
> 2026-10-17T11:47:24.235808+00:00 ofbiz-ec2-or kernel: CPU: 2 UID: 100999 PID: 1648549 Comm: postgres Not tainted 7.0.0-1014-aws #14~24.04.1-Ubuntu
> PREEMPTLAZY
> 2026-10-17T11:47:24.235813+00:00 ofbiz-ec2-or kernel: Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
> ...
> ...
> 2026-10-17T11:47:24.239755+00:00 ofbiz-ec2-or kernel: memory: usage 307200kB, limit 307200kB, failcnt 1879
> 2026-10-17T11:47:24.239771+00:00 ofbiz-ec2-or kernel: swap: usage 0kB, limit 0kB, failcnt 0
> 2026-10-17T11:47:24.239773+00:00 ofbiz-ec2-or kernel: Memory cgroup stats for
> /system.slice/docker-59d2f448671fcb358419dfe26eced00b69cebe3e7edfbc9e16f3fb3a0f0746b9.scope:
>
>
> In the syslog snippet above we see mention of the 'oom-killer'. This is the kernel killing processes when Out Of Memory.
> The log shows that the process was using 307200KB (approx 300MB) which was at the limit, triggering the process to be killed.
>
> I have run the /home/ofbizdocker/pull-and-restart.sh script as a way to tear down and restart the demos. The demos appear to have restarted and are
> operational again.
>
> If we see this type of problem occuring again we will need to decide if the DB memory limit is too low, or whether we should be investigating some
> other aspect of OFBiz.
>
> Thanks
>
> Dan.
>
>
> On Sat, 17 Oct 2026 at 18:34, Daniel Watford <da...@foomoo.co.uk> wrote:
>
> Hi Jaques,
>
> I'm also taking a look. There appear to be a huge number of logs. So many that it is difficult to find the relevant text.
>
> There doesn't appear to be any issues with lack of disk space.
>
> I'll investigate by shutting the demos down and then bringing them up one by one, checking logs as I go.
>
> Thanks
>
> Dan.
>
> On Sat, 17 Oct 2026 at 17:31, Jacques Le Roux via dev <de...@ofbiz.apache.org> wrote:
>
> Hi,
>
> Between 16:45 UTC and "now", OFBiz demos are slow (or even very slow) or down.
> Trunk is by far the much affected, which is weird... I'll have a look...
>
> Jacques
>
>
>
>
> --
> Daniel Watford
>
>
>
> --
> Daniel Watford
Re: OFBiz demos are very slow or down
Posted by Daniel Watford <da...@foomoo.co.uk>.
Summary: Postgres database exceeded its 300 MB memory limit, causing the
kernel to kill the process. Upon restart, the database failed to recover
from corruption.
Resolved by: Tearing down and restarting demos. This caused the databases
to reset.
To consider: Perhaps the 300 MB limit for the database process is too low?
Or perhaps some run-away process in object caused excessive database use?
Hi Jacques,
I took a look at the standard output and standard error logs for
ofbiz-trunk and found there was a huge amount, suggesting some sort of
cycle.
Filtering the logs for the Postgres database, I used the command:
docker compose logs db | less
Extract from the logs:
db-1 | 2026-10-17 02:35:56.952 UTC [64] LOG: database system was shut
down at 2026-10-17 02:35:56 UTC
db-1 | 2026-10-17 02:35:56.962 UTC [1] LOG: database system is ready to
accept connections
db-1 | 2026-10-17 11:47:24.253 UTC [1] LOG: server process (PID 1747) was
terminated by signal 9: Killed
db-1 | 2026-10-17 11:47:24.253 UTC [1] DETAIL: Failed process was
running: SELECT PTYACCPREF.PARTY_ID, PTYACCPREF.BASE_CURRENCY_UOM_ID,
PTYGROUP.GROUP_NAME, PTYROLE.ROLE_TYPE_ID FROM
(public.PARTY_ACCTG_PREFERENCE PTYACCPREF INNER JOIN public.PARTY_GROUP
PTYGROUP ON PTYACCPREF.PARTY_ID = PTYGROUP.PARTY_ID) INNER JOIN
public.PARTY_ROLE PTYROLE ON PTYACCPREF.PARTY_ID = PTYROLE.PARTY_ID WHERE
(((PTYROLE.ROLE_TYPE_ID = $1)))
db-1 | 2026-10-17 11:47:24.253 UTC [1] LOG: terminating any other active
server processes
db-1 | 2026-10-17 11:47:24.253 UTC [2369] WARNING: terminating connection
because of crash of another server process at character 245
db-1 | 2026-10-17 11:47:24.253 UTC [2369] DETAIL: The postmaster has
commanded this server process to roll back the current transaction and
exit, because another server process exited abnormally and possibly
corrupted shared memory.
db-1 | 2026-10-17 11:47:24.253 UTC [2369] HINT: In a moment you should be
able to reconnect to the database and repeat your command.
db-1 | 2026-10-17 11:47:24.253 UTC [2357] WARNING: terminating connection
because of crash of another server process
db-1 | 2026-10-17 11:47:24.253 UTC [2357] DETAIL: The postmaster has
commanded this server process to roll back the current transaction and
exit, because another server process exited abnormally and possibly
corrupted shared memory.
It appears that the database was killed at 11:47:24.253 UTC. The question
is why....
Searching in /var/log/syslog for 2026-10-17T11:47 we find:
2026-10-17T11:47:24.235788+00:00 ofbiz-ec2-or kernel: postgres invoked
oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
2026-10-17T11:47:24.235808+00:00 ofbiz-ec2-or kernel: CPU: 2 UID: 100999
PID: 1648549 Comm: postgres Not tainted 7.0.0-1014-aws #14~24.04.1-Ubuntu
PREEMPTLAZY
2026-10-17T11:47:24.235813+00:00 ofbiz-ec2-or kernel: Hardware name: Xen
HVM domU, BIOS 4.11.amazon 08/24/2006
...
...
2026-10-17T11:47:24.239755+00:00 ofbiz-ec2-or kernel: memory: usage
307200kB, limit 307200kB, failcnt 1879
2026-10-17T11:47:24.239771+00:00 ofbiz-ec2-or kernel: swap: usage 0kB,
limit 0kB, failcnt 0
2026-10-17T11:47:24.239773+00:00 ofbiz-ec2-or kernel: Memory cgroup stats
for
/system.slice/docker-59d2f448671fcb358419dfe26eced00b69cebe3e7edfbc9e16f3fb3a0f0746b9.scope:
In the syslog snippet above we see mention of the 'oom-killer'. This is the
kernel killing processes when Out Of Memory.
The log shows that the process was using 307200KB (approx 300MB) which was
at the limit, triggering the process to be killed.
I have run the /home/ofbizdocker/pull-and-restart.sh script as a way to
tear down and restart the demos. The demos appear to have restarted and are
operational again.
If we see this type of problem occuring again we will need to decide if the
DB memory limit is too low, or whether we should be investigating some
other aspect of OFBiz.
Thanks
Dan.
On Sat, 17 Oct 2026 at 18:34, Daniel Watford <da...@foomoo.co.uk> wrote:
> Hi Jaques,
>
> I'm also taking a look. There appear to be a huge number of logs. So many
> that it is difficult to find the relevant text.
>
> There doesn't appear to be any issues with lack of disk space.
>
> I'll investigate by shutting the demos down and then bringing them up one
> by one, checking logs as I go.
>
> Thanks
>
> Dan.
>
> On Sat, 17 Oct 2026 at 17:31, Jacques Le Roux via dev <
> dev@ofbiz.apache.org> wrote:
>
>> Hi,
>>
>> Between 16:45 UTC and "now", OFBiz demos are slow (or even very slow) or
>> down.
>> Trunk is by far the much affected, which is weird... I'll have a look...
>>
>> Jacques
>>
>>
>>
>
> --
> Daniel Watford
>
--
Daniel Watford
Re: OFBiz demos are very slow or down
Posted by Daniel Watford <da...@foomoo.co.uk>.
Hi Jaques,
I'm also taking a look. There appear to be a huge number of logs. So many
that it is difficult to find the relevant text.
There doesn't appear to be any issues with lack of disk space.
I'll investigate by shutting the demos down and then bringing them up one
by one, checking logs as I go.
Thanks
Dan.
On Sat, 17 Oct 2026 at 17:31, Jacques Le Roux via dev <de...@ofbiz.apache.org>
wrote:
> Hi,
>
> Between 16:45 UTC and "now", OFBiz demos are slow (or even very slow) or
> down.
> Trunk is by far the much affected, which is weird... I'll have a look...
>
> Jacques
>
>
>
--
Daniel Watford
Re: OFBiz demos are very slow or down
Posted by Jacques Le Roux via dev <de...@ofbiz.apache.org>.
Le 17/10/2026 à 18:29, Jacques Le Roux via dev a écrit :
> Hi,
>
> Between 16:45 UTC and "now", OFBiz demos are slow (or even very slow) or down.
> Trunk is by far the much affected, which is weird... I'll have a look...
>
> Jacques
There is a DB issue in trunk, but not in stable. Whatever both are slow. Better wait their restart this night.
Jacques