You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by TC <th...@gmail.com> on 2019/04/17 23:20:02 UTC
Kahadb journal logs only deleted when activemq restarts
Hi,
I have broker that runs with increasing journal logs file with mKahadb store
for each destination.
One of the queue - journal logs are keep increasing overtime.
The strange part is that - everytime I restart activemq, these journal log
files get deleted.
To me, it looks like the message within these journal log files are not
referenced anymore - it should be get deleted during the gc cycle.
The question - what could caused this problem ?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Kahadb journal logs only deleted when activemq restarts
Posted by TC <th...@gmail.com>.
I strongly believe these MyEventQueue1 and MyEventQueue2 queues messages
are consumed successfully .
As indicated, there are many duplicates messages detected for the same queue
which resulted in DLQ. I deployed the broker DLQ plugin
(discardingDLQBrokerPlugin) to discard all the messages.
I did stopped the broker DLQ plugin and inspect sample of these DLQ
messages; all the headers (dlqDeliveryFailureCause) with details :
java.lang.Throwable: duplicate from store for queue. These sample messages
did indeed processed/consumed by the application - hence i believe these
messages are consumed successfully. I do not think there is any expiration
set on these messages.
The symptom I see - messages are not there however the journal logs say
otherwise (and it is increasing). Occasionally a few data files get removed.
The journal logs files get deleted at a very slow rate compared with the
rate journal logs file get created. This behavior is observed with
concurrentStoreAndDispatchQueues is set to true. When I restart the broker,
all these journal logs file (db-xxx,log) get removed/deleted.
When I switch the mKahaDB for the 2 queues to
concurrentStoreAndDispatchQueues to false, I can see the journal log files
get deleted, i.e only 1 journal log file at any point of time... and the
number - db-<the number>.log get increased fairly quick - but at any point
of time, there is only 1 journal log file.
I believe the message workload is too fast to be handle by mKahaDB storage -
need to test out other options in KahaDB to get the best result - looking at
indexCacheSize, indexWriteBatchSize, compactAcksIgnoresStoreGrowth and
possibly setting journalDiskSyncStrategy=“never" to see the results on these
2 queues.
Do share out if you have any idea.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Kahadb journal logs only deleted when activemq restarts
Posted by Tim Bain <tb...@alumni.duke.edu>.
So an earlier post indicated that the MyEventQueue1 and MyEventQueue2
queues were both(?) retaining messages. Do you believe that those messages
were all consumed successfully, and so the queues were empty but the KahaDB
data files were not being removed? Or were there a significant number of
unconsumed messages on those queues at the time of the broker restart? Your
most recent message seems to indicate the latter, though nothing you wrote
before that gave any indication that that might be the case.
Also, what message expiration settings are in use, and what DLQ behavior
have you configured?
Tim
On Sun, Apr 21, 2019, 7:56 PM TC <th...@gmail.com> wrote:
> I managed to clear the journal logs on those queues mentioned.
> The daily (earlier) journal logs file size is about 10GB.
>
> The journal logs only get cleared when concurrentStoreAndDispatchQueues is
> set to false.
>
> However, my producer application ( runs another ActiveMQ server), active
> non-durable subscribers messages get piled up due to the slowness in
> message consumption on my end.
>
> I am running with 16 core cpu Intel on Linux server with 25GB heap size,
> and
> i am using SSD disk on mKahadb.
>
> What is my option available - besides setting off persistence ?
>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Kahadb journal logs only deleted when activemq restarts
Posted by TC <th...@gmail.com>.
I managed to clear the journal logs on those queues mentioned.
The daily (earlier) journal logs file size is about 10GB.
The journal logs only get cleared when concurrentStoreAndDispatchQueues is
set to false.
However, my producer application ( runs another ActiveMQ server), active
non-durable subscribers messages get piled up due to the slowness in
message consumption on my end.
I am running with 16 core cpu Intel on Linux server with 25GB heap size, and
i am using SSD disk on mKahadb.
What is my option available - besides setting off persistence ?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Kahadb journal logs only deleted when activemq restarts
Posted by Tim Bain <tb...@alumni.duke.edu>.
On Fri, Apr 19, 2019, 6:34 PM TC <th...@gmail.com> wrote:
>
> "I wasn't aware of any reason that the logging config and debugging
> techniques described on that page would not work properly with mKahaDB. Can
> you please elaborate on the failure you referenced?"
>
> - Added the log4j.properties with the following
> log4j.appender.kahadb=org.apache.log4j.RollingFileAppender
> log4j.appender.kahadb.file=/myowndirectory/kahadb.log
> /data/kahadb.log log4j.appender.kahadb.maxFileSize=1024KB
> log4j.appender.kahadb.maxBackupIndex=5 log4j.appender.kahadb.append=true
> log4j.appender.kahadb.layout=org.apache.log4j.PatternLayout
> log4j.appender.kahadb.layout.ConversionPattern=%d [%-15.15t] %-5p
> %-30.30c{1} - %m%n
> log4j.logger.org.apache.activemq.store.kahadb.MessageDatabase=TRACE, kahadb
>
> - did not restart the ActiveMQ - rather than waited for 10 minutes or so
>
Do you have any information to indicate that this process (waiting 10
minutes) will cause the ActiveMQ process to reload its Log4J config file?
The page Chris referenced describes two procedures (the only two I'm aware
of), neither of which is to wait for a set amount of time. Try using the
JMX procedure and see if that works with your mKahaDB setup.
- there is no kahadb.log file exist in the directory.
>
> The ActiveMQ version i am using 5.14.4
>
> " Also, Sunil rightly pointed out that the behavior you've described can be
> caused by uncommitted transactions. Are transactions in use anywhere, and
> if so have you explored the possibility that they might not be getting
> committed? "
>
> - What is uncommitted transactions ? Based on mKahaDB with per destination
> journal logs , the increasing logs are increasing are the 2 queues
> mentioned
> earlier, i.e. forwarded from topic using VirtualDestination. These queue
> are processed by a Node.js stomp client which basically inserting the
> message data into a NoSQL database - there is no transactions per say in
> the
> code itself.
>
Transactions in this context are a JMS concept:
http://activemq.apache.org/how-do-transactions-work. Although they can be
tied to a SQL transaction, they don't have to be, and wouldn't be since
you're using mKahaDB. Since your consumers all use STOMP, then there are no
transactions in use, because STOMP doesn't support transactions. So we'll
have to see what you learn from following the process described on the page
Chris provided.
Tim
>
Re: Kahadb journal logs only deleted when activemq restarts
Posted by TC <th...@gmail.com>.
"I wasn't aware of any reason that the logging config and debugging
techniques described on that page would not work properly with mKahaDB. Can
you please elaborate on the failure you referenced?"
- Added the log4j.properties with the following
log4j.appender.kahadb=org.apache.log4j.RollingFileAppender
log4j.appender.kahadb.file=/myowndirectory/kahadb.log
/data/kahadb.log log4j.appender.kahadb.maxFileSize=1024KB
log4j.appender.kahadb.maxBackupIndex=5 log4j.appender.kahadb.append=true
log4j.appender.kahadb.layout=org.apache.log4j.PatternLayout
log4j.appender.kahadb.layout.ConversionPattern=%d [%-15.15t] %-5p
%-30.30c{1} - %m%n
log4j.logger.org.apache.activemq.store.kahadb.MessageDatabase=TRACE, kahadb
- did not restart the ActiveMQ - rather than waited for 10 minutes or so
- there is no kahadb.log file exist in the directory.
The ActiveMQ version i am using 5.14.4
" Also, Sunil rightly pointed out that the behavior you've described can be
caused by uncommitted transactions. Are transactions in use anywhere, and
if so have you explored the possibility that they might not be getting
committed? "
- What is uncommitted transactions ? Based on mKahaDB with per destination
journal logs , the increasing logs are increasing are the 2 queues mentioned
earlier, i.e. forwarded from topic using VirtualDestination. These queue
are processed by a Node.js stomp client which basically inserting the
message data into a NoSQL database - there is no transactions per say in the
code itself.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Kahadb journal logs only deleted when activemq restarts
Posted by Tim Bain <tb...@alumni.duke.edu>.
I wasn't aware of any reason that the logging config and debugging
techniques described on that page would not work properly with mKahaDB. Can
you please elaborate on the failure you referenced?
Also, Sunil rightly pointed out that the behavior you've described can be
caused by uncommitted transactions. Are transactions in use anywhere, and
if so have you explored the possibility that they might not be getting
committed?
What version of ActiveMQ are you using?
Tim
On Fri, Apr 19, 2019, 5:18 AM TC <th...@gmail.com> wrote:
> Based on the link
>
> https://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html
> I tried to setup the journal log to see which the reasons for the missing
> logs. But failed -my setup is using mKahadb rather than Kahadb.
>
> I only knows which queue that has lots of journal logs - in my case - it is
> a queue that get forwarded from a topic - sample of the activemq.xml
>
> <compositeTopic name="MyEventTopic">
> <forwardTo>
> <queue physicalName="MyEventQueue1" />
> <queue physicalName="MyEventQueue2" />
> </forwardTo>
> </compositeTopic>
>
> The queue that fills up journal logs in mKahadb are MyEventQueue1 and
> MyEventQueue2.
> And in my activemq.log there are many many many duplicate message detected
> for MyEventQueue1 and MyEventQueue2.
>
> And these journal logs get deleted everytime I restart ActiveMQ broker.
>
>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Kahadb journal logs only deleted when activemq restarts
Posted by TC <th...@gmail.com>.
Based on the link
https://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html
I tried to setup the journal log to see which the reasons for the missing
logs. But failed -my setup is using mKahadb rather than Kahadb.
I only knows which queue that has lots of journal logs - in my case - it is
a queue that get forwarded from a topic - sample of the activemq.xml
<compositeTopic name="MyEventTopic">
<forwardTo>
<queue physicalName="MyEventQueue1" />
<queue physicalName="MyEventQueue2" />
</forwardTo>
</compositeTopic>
The queue that fills up journal logs in mKahadb are MyEventQueue1 and
MyEventQueue2.
And in my activemq.log there are many many many duplicate message detected
for MyEventQueue1 and MyEventQueue2.
And these journal logs get deleted everytime I restart ActiveMQ broker.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Kahadb journal logs only deleted when activemq restarts
Posted by Christopher Shannon <ch...@gmail.com>.
This page has some info that might help you:
https://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html
On Wed, Apr 17, 2019 at 7:43 PM TC <th...@gmail.com> wrote:
> Hi,
>
> I have broker that runs with increasing journal logs file with mKahadb
> store
> for each destination.
>
> One of the queue - journal logs are keep increasing overtime.
>
> The strange part is that - everytime I restart activemq, these journal log
> files get deleted.
> To me, it looks like the message within these journal log files are not
> referenced anymore - it should be get deleted during the gc cycle.
>
> The question - what could caused this problem ?
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Kahadb journal logs only deleted when activemq restarts
Posted by Sunil Vishwanath <su...@icloud.com.INVALID>.
Usually this is caused by uncommitted transactions
Sent from my iPhone
> On Apr 17, 2019, at 7:20 PM, TC <th...@gmail.com> wrote:
>
> Hi,
>
> I have broker that runs with increasing journal logs file with mKahadb store
> for each destination.
>
> One of the queue - journal logs are keep increasing overtime.
>
> The strange part is that - everytime I restart activemq, these journal log
> files get deleted.
> To me, it looks like the message within these journal log files are not
> referenced anymore - it should be get deleted during the gc cycle.
>
> The question - what could caused this problem ?
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html