Issues with the Artemis 2.40+ journal

[Alexander Milovidov <mi...@gmail.com>]

Hi all,

On one of our Artemis clusters, we are observing events such as:

AMQ222153 WARN Cannot locate record for message id = 343528886489 on Journal
AMQ142005 WARN Inconsistency during compacting: Delete record being read on
an existent record (id=343529098539)

The cluster is configured with primary-backup replication and a ZooKeeper
quorum.

When restarting the active server in the cluster, issues occurred: slow
startup, the critical analyzer triggering during startup, and message
accumulation in queues. The issue was resolved by restarting the entire
cluster, but the journal-related errors persisted. After that, the journal
was recreated, but after some time, journal-related events began appearing
in the logs again.

These events started appearing after upgrading Artemis from version 2.39.0
to 2.53.0. The problem also reproduces in version 2.55.0.
While Artemis is running with these warnings, journal files accumulate on
the server, and disk space usage periodically increases and then decreases
(likely due to journal compaction attempts). After a very long period of
running with these problems, upon restart, Artemis crashes with an
OutOfMemoryError and fails to start.

Load details. This Artemis instance is used to publish data to a single
topic address and to forward messages to consumer queues via divert. There
are approximately 100 divert configurations of the following kind on the
server:

<divert name="APP1.SUBSCRIPTION">
  <address>TOPIC</address>
  <forwarding-address>APP1.SUB.QUEUE</forwarding-address>
  <exclusive>false</exclusive>
  <filter string="key = 'value'"/>
</divert>

About 80% of the diverts use filters. All messages are consumed; however,
some messages are not processed immediately but only after a small amount
of accumulation and subsequent paging. The problem reproduces only when
reading messages from paging.

The issue reproduces only with production messages. We were unable to
reproduce it with synthetic messages of the same size. We attempted to
create a reproducible example in an isolated environment, but
unfortunately, we have not been able to produce such a configuration.

Approximately 2.5 million messages per day are published to the topic, with
an average message size of about 9–10 KB plus headers of 1.5–2 KB (roughly
45 properties). Publishing rate ranges from 180 to 600 messages per second.
The problem is observed starting at a load of around 200 messages per
second. According to the logs, a total of roughly 500–1000 GB of data
passes through the cluster per day.

Current journal settings:

<journal-type>ASYNCIO</journal-type>
<journal-datasync>true</journal-datasync>
<journal-device-block-size>4096</journal-device-block-size>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-file-size>50M</journal-file-size>
<journal-buffer-timeout>88000</journal-buffer-timeout>

We attempted to change the settings - increasing journal-file-size and
reducing compaction frequency (increasing journal-compact-min-files,
increasing journal-compact-percentage) - but this had no effect on the
problem.

It is possible that the issue is related to the improvement made as part of
ARTEMIS-5305.

Is there any settings which we should try to change? Can I create a Jira?

--
Regards,
Alexander