Kahadb log files increase until reaches its limit

[Rafael Pimenta <ra...@gmail.com>]

Hi Dears,
I'm using ActiveMQ 5.15.9 and after about one month our customers started
to complain about the Kahadb folder increase until reaching its limit and
crash the system. Then they delete the folder to recover the system, but we
are losing data in this process.
We activate the kahadb log and seems that some old messages are getting
stuck for some reason and do not allow the ActiveMQ to delete the db-x.log
files:

*2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244, 243],
245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247], 252=[140,
108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141], 256=[256,
250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259], 260=[259,
260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262, 254,
263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263], 270=[159],
271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274, 271],
275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277, 278],
279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282], 283=[159],
284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288, 287],
289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297], 298=[174,
159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302], 303=[302,
303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308, 306,
287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311], 313=[312,
313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326, 278,
294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329, 330],
331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251, 333,
253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337, 339],
342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347], 349=[348,
337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302, 351,
271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21], 355=[353,
355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356, 358],
359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361, 362],
363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367], 370=[369,
370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372, 373],
374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376, 377],
378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383], 385=[384,
385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337, 385,
386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384, 388,
389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337, 274,
243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34], 399=[380,
397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37, 38],
406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407], 410=[409,
410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401, 409,
389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421], 422=[385,
419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394, 414],
424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422, 425,
426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400, 424,
403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414, 431],
432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435], 436=[49,
51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52, 40],
440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446], 447=[438,
446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449], 450=[53,
54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394, 414],
453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459], 460=[49, 53,
57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462], 463=[58,
60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466, 467],
468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470, 471],
472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474, 475],
476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483, 484],
485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64, 66],
490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493], 494=[493,
494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490, 459,
446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511], 512=[512,
438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76], 517=[515,
517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500, 438,
503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511], 520=[519,
520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525, 526],
527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529, 531],
532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517, 534,
535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71], 540=[538,
540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544, 543],
545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531, 548,
535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554], 555=[97],
556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558], 559=[64, 68,
70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70, 55,
105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568, 569],
570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577, 561,
554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588], 589=[211],
590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596, 597],
598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597, 599,
600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602, 586,
604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609, 611,
586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599, 600],
614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615, 599,
616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618], 619=[618,
619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621, 622],
623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625, 616,
621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615, 619,
622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631, 634,
635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622], 639=[243]} |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
contained ack(s) refer to referenced file: [244, 243] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
contained ack(s) refer to referenced file: [123] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
contained ack(s) refer to referenced file: [244, 246] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
contained ack(s) refer to referenced file: [246, 247] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,564 | WARN  | *Stealing link for clientId
TR_VP-018:30246 From Connection Transport Connection to: tcp://
172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker |
ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
contained ack(s) refer to referenced file: [123] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
contained ack(s) refer to referenced file: [249, 247] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
contained ack(s) refer to referenced file: [163, 211, 249, 250, 234, 237] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
contained ack(s) refer to referenced file: [250, 243, 251, 247] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
contained ack(s) refer to referenced file: [123] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
contained ack(s) refer to referenced file: [249, 247] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
contained ack(s) refer to referenced file: [163, 211, 249, 250, 234, 237] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
contained ack(s) refer to referenced file: [250, 243, 251, 247] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
contained ack(s) refer to referenced file: [140, 108, 97] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
contained ack(s) refer to referenced file: [249, 253, 251] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
contained ack(s) refer to referenced file: [253, 254] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
contained ack(s) refer to referenced file: [140, 141] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker not removing data file: 252 as contained ack(s) refer to
referenced file: [140, 108, 97] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
contained ack(s) refer to referenced file: [249, 253, 251] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
contained ack(s) refer to referenced file: [253, 254] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
*2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
contained ack(s) refer to referenced file: [140, 141] |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker

We also use durable topics as the devices that subscribe to these topics
and send MQTT messages to others work in a very bad network environment,
this is the way we found to avoid data loss while offline.
I think the problem is related to durable topics with offline
subscriptions, but how can I confirm this assumption, eg which topics? What
signatures?

I much appreciate any thoughts!
Thanks in advance

Re: Kahadb log files increase until reaches its limit

[Mark Johnson <sp...@gmail.com>]

Hi Rafael,

Our testing has shown that multi does use more resources for the same
throughput; more cpu, ram and storage, but not significantly more unless
you are already at the limit.
We have configured one-kaha-per-queue so it does improve the storage
reclaim when for whatever reason the reclaim process decides not to reclaim
certain journals.

We still have not figured out why the reclaim is deciding not to reclaim;
as far as we can tell all msg reads are ACK'd, but it still happens in
production that purging some queues has almost zero impact on the storage
whilst purging others will reclaim 90% of storage.

The current working hypothesis is that there is some interaction between
Camel transaction manager, JDBC commit failures, and rolling back msg reads.

Best wishes in getting your config sorted.

On Wed, 5 Oct 2022 at 14:20, Rafael Pimenta <ra...@gmail.com> wrote:

> As suggested by Mark Johnson I'm evaluating a multi-kahadb setup:
>         <persistenceAdapter>
>         <mKahaDB directory="${activemq.base}/data/kahadb">
>             <filteredPersistenceAdapters>
>             <!-- kahaDB per destinations -->
>             <filteredKahaDB perDestination="true">
>                 <persistenceAdapter>
>                 <kahaDB journalMaxFileLength="1mb"/>
>                 </persistenceAdapter>
>             </filteredKahaDB>
>             </filteredPersistenceAdapters>
>         </mKahaDB>
>         </persistenceAdapter>
>
> But, after that, I'm facing the following warning:
> 2022-10-04 18:05:23,555 | WARN  | Destination name is longer than
> 'MaximumFileNameLength' system property, potential problem with recovery
> can result from name truncation. |
> org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter | ActiveMQ
> Transport: tcp:///192.168.100.123:34926@1883
>
> Where can I change the MaximumFileNameLength to be greater than 64
> characters?
> I saw this ticket related to the underlying problem, but I can't see where
> to change that configuration:
> https://issues.apache.org/jira/browse/AMQ-4183
>
> Att,
> *Rafael Pimenta*
>
> Em seg., 3 de out. de 2022 às 10:34, Rafael Pimenta <
> rafa.spimenta@gmail.com>
> escreveu:
>
> > Hello Marco,
> > First of All, Thanks for your reply and detailed explanation.
> >
> > To avoid increasing the KahaDB with DLQ queues, we disable the DLQ, so
> > expired messages will be discarded automatically:
> >
> > <deadLetterStrategy>
> >   <sharedDeadLetterStrategy processExpired="false" />
> > </deadLetterStrategy>
> >
> > Because of that configuration, I assume that the problems fall on durable
> > topics.
> > As we know the number of durable topics must be created every time our
> > system start, I'll study virtual topics and check if it is suitable for
> our
> > purposes. I'll let you ASAP.
> >
> > Thanks!
> > *Rafael Pimenta*
> >
> >
> >
> >
> > Em dom., 2 de out. de 2022 às 08:08, Marco Bettiol <ja...@gmail.com>
> > escreveu:
> >
> >> Hi Rafael,
> >> I’m just an ActiveMQ classic user.
> >>
> >> I will report the behavior of ActiveMQ I experienced.
> >>
> >> As far as I know the ActiveMQ classic journal is composed of append only
> >> files.
> >> Each file should be by default 32MB in size (this can be changed, I
> think
> >> is controlled by journalLogFileSize)
> >> A journal file gets released only when ALL the messages within are
> >> consumed.
> >> Furthermore, messages form different queues are appended to and mixed
> >> within the same file until the maximum size gets reached.
> >>
> >> As consequence if you have unprocessed messages (DLQ or offline
> consumers)
> >> the used storage will increase, usually in the same order of magnitude
> as
> >> the total traffic served by the broker: a small percentage of unconsumed
> >> messages will prevent journal file cleanup.
> >>
> >> For example an unprocessed 1KByte message will retain 32MB on disk.
> >>
> >> As far as I know there is no building functionality to perform journal
> >> compaction.
> >>
> >> In our case the developed internally a small application to offload
> >> unprocessed messages to an external database/ requeue DLQ messages to
> the
> >> same DLQ in order to free storage.
> >>
> >> In my experience persistent Topics are quite hard to monitor, if the
> list
> >> of subscribers is not dynamic  (known when application starts) you could
> >> think to use VirtualTopics so that you will get back to a normal queue
> >> uses
> >> case for each consumer.
> >>
> >> Regards,
> >> Marc
> >>
> >> Il giorno gio 29 set 2022 alle ore 23:06 Rafael Pimenta <
> >> rafa.spimenta@gmail.com> ha scritto:
> >>
> >> > Hi Dears,
> >> > I'm using ActiveMQ 5.15.9 and after about one month our customers
> >> started
> >> > to complain about the Kahadb folder increase until reaching its limit
> >> and
> >> > crash the system. Then they delete the folder to recover the system,
> >> but we
> >> > are losing data in this process.
> >> > We activate the kahadb log and seems that some old messages are
> getting
> >> > stuck for some reason and do not allow the ActiveMQ to delete the
> >> db-x.log
> >> > files:
> >> >
> >> > *2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244,
> 243],
> >> > 245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
> >> > 250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247],
> 252=[140,
> >> > 108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141],
> 256=[256,
> >> > 250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259],
> >> 260=[259,
> >> > 260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262,
> >> 254,
> >> > 263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
> >> > 268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263],
> 270=[159],
> >> > 271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274,
> >> 271],
> >> > 275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277,
> >> 278],
> >> > 279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282],
> >> 283=[159],
> >> > 284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288,
> >> 287],
> >> > 289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
> >> > 294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297],
> >> 298=[174,
> >> > 159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302],
> >> 303=[302,
> >> > 303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308,
> >> 306,
> >> > 287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311],
> >> 313=[312,
> >> > 313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
> >> > 318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
> >> > 323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326,
> 278,
> >> > 294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329,
> >> 330],
> >> > 331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251,
> >> 333,
> >> > 253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337,
> >> 339],
> >> > 342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
> >> > 346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347],
> 349=[348,
> >> > 337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302,
> 351,
> >> > 271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21],
> 355=[353,
> >> > 355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356,
> 358],
> >> > 359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
> >> > 266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361,
> 362],
> >> > 363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
> >> > 367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367],
> 370=[369,
> >> > 370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372,
> 373],
> >> > 374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376,
> >> 377],
> >> > 378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
> >> > 381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383],
> 385=[384,
> >> > 385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337,
> 385,
> >> > 386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384,
> 388,
> >> > 389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
> >> > 392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
> >> > 396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337,
> >> 274,
> >> > 243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34],
> >> 399=[380,
> >> > 397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
> >> > 401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37,
> 38],
> >> > 406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407],
> >> 410=[409,
> >> > 410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401,
> 409,
> >> > 389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
> >> > 419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421],
> 422=[385,
> >> > 419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394,
> 414],
> >> > 424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422,
> 425,
> >> > 426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400,
> 424,
> >> > 403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414,
> 431],
> >> > 432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435],
> >> 436=[49,
> >> > 51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52,
> 40],
> >> > 440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
> >> > 444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446],
> 447=[438,
> >> > 446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449],
> 450=[53,
> >> > 54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394,
> 414],
> >> > 453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
> >> > 457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459],
> 460=[49,
> >> 53,
> >> > 57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462],
> >> 463=[58,
> >> > 60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466,
> >> 467],
> >> > 468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470,
> >> 471],
> >> > 472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474,
> >> 475],
> >> > 476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
> >> > 480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483,
> >> 484],
> >> > 485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64,
> >> 66],
> >> > 490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493],
> 494=[493,
> >> > 494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
> >> > 498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
> >> > 501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490,
> >> 459,
> >> > 446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
> >> > 508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511],
> >> 512=[512,
> >> > 438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76],
> 517=[515,
> >> > 517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500,
> >> 438,
> >> > 503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511],
> 520=[519,
> >> > 520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
> >> > 524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525,
> >> 526],
> >> > 527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529,
> 531],
> >> > 532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517,
> 534,
> >> > 535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71],
> >> 540=[538,
> >> > 540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544,
> 543],
> >> > 545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531,
> 548,
> >> > 535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
> >> > 552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554],
> 555=[97],
> >> > 556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558],
> 559=[64,
> >> 68,
> >> > 70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70,
> 55,
> >> > 105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
> >> > 566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568,
> >> 569],
> >> > 570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
> >> > 574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577,
> 561,
> >> > 554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
> >> > 581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
> >> > 585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588],
> >> 589=[211],
> >> > 590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
> >> > 594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596,
> >> 597],
> >> > 598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597,
> 599,
> >> > 600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602,
> 586,
> >> > 604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
> >> > 608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609,
> 611,
> >> > 586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599,
> 600],
> >> > 614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615,
> 599,
> >> > 616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618],
> 619=[618,
> >> > 619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621,
> >> 622],
> >> > 623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625,
> 616,
> >> > 621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615,
> 619,
> >> > 622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
> >> > 633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631,
> 634,
> >> > 635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622],
> >> 639=[243]} |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
> >> > contained ack(s) refer to referenced file: [244, 243] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
> >> > contained ack(s) refer to referenced file: [123] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
> >> > contained ack(s) refer to referenced file: [244, 246] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
> >> > contained ack(s) refer to referenced file: [246, 247] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,564 | WARN  | *Stealing link for clientId
> >> > TR_VP-018:30246 From Connection Transport Connection to: tcp://
> >> > 172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker
> |
> >> > ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
> >> > contained ack(s) refer to referenced file: [123] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
> >> > contained ack(s) refer to referenced file: [249, 247] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
> >> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
> >> 237] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
> >> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
> >> > contained ack(s) refer to referenced file: [123] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
> >> > contained ack(s) refer to referenced file: [249, 247] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
> >> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
> >> 237] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
> >> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
> >> > contained ack(s) refer to referenced file: [140, 108, 97] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
> >> > contained ack(s) refer to referenced file: [249, 253, 251] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
> >> > contained ack(s) refer to referenced file: [253, 254] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
> >> > contained ack(s) refer to referenced file: [140, 141] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker not removing data file: 252 as contained ack(s)
> refer
> >> to
> >> > referenced file: [140, 108, 97] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
> >> > contained ack(s) refer to referenced file: [249, 253, 251] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
> >> > contained ack(s) refer to referenced file: [253, 254] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
> >> > contained ack(s) refer to referenced file: [140, 141] |
> >> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> >> > Checkpoint Worker
> >> >
> >> > We also use durable topics as the devices that subscribe to these
> topics
> >> > and send MQTT messages to others work in a very bad network
> environment,
> >> > this is the way we found to avoid data loss while offline.
> >> > I think the problem is related to durable topics with offline
> >> > subscriptions, but how can I confirm this assumption, eg which topics?
> >> What
> >> > signatures?
> >> >
> >> > I much appreciate any thoughts!
> >> > Thanks in advance
> >> >
> >>
> >
>

Re: Kahadb log files increase until reaches its limit

[Rafael Pimenta <ra...@gmail.com>]

After a while of using the ActiveMQ with mKahadb it crashes with the
following errors:

2022-10-05 14:14:06,735 | WARN  | Error subscribing to
/DvxSrv/12/Mine/Sampling |
org.apache.activemq.transport.mqtt.strategy.AbstractMQTTSubscriptionStrategy
| ActiveMQ Transport: tcp:///10.26.163.157:40380@1883
*java.lang.OutOfMemoryError: Java heap space*
    at
org.apache.activemq.store.kahadb.disk.journal.Journal.recoveryCheck(Journal.java:499)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at
org.apache.activemq.store.kahadb.disk.journal.Journal.start(Journal.java:334)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at
org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:458)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:502)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at
org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:306)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at
org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:221)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.9.jar:5.15.9]
    at
org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:232)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client-5.15.9.jar:5.15.9]
    at
org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.startAdapter(MultiKahaDBPersistenceAdapter.java:212)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at
org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.getMatchingPersistenceAdapter(MultiKahaDBPersistenceAdapter.java:205)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at
org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.createTopicMessageStore(MultiKahaDBPersistenceAdapter.java:232)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:99)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:626)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.jmx.ManagedTopicRegion.createDestination(ManagedTopicRegion.java:56)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:155)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:339)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:242)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:174)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.region.AbstractRegion.lookup(AbstractRegion.java:558)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:345)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:188)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:418)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:104)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:131)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:104)[activemq-broker-5.15.9.jar:5.15.9]
    at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:104)[activemq-broker-5.15.9.jar:5.15.9]

2022-10-05 14:14:12,329 | WARN  | Exception encountered during context
initialization - cancelling refresh attempt:
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'secHandlerCollection' defined in class path resource
[jetty.xml]: Cannot create inner bean
'org.eclipse.jetty.webapp.WebAppContext#9b6d9f7' of type
[org.eclipse.jetty.webapp.WebAppContext] while setting bean property
'handlers' with key [1]; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'org.eclipse.jetty.webapp.WebAppContext#9b6d9f7' defined in
class path resource [jetty.xml]: Initialization of bean failed; nested
exception is java.lang.OutOfMemoryError: *GC overhead limit exceeded* |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain

2022-10-05 14:14:14,454 | INFO  | Stopping BrokerService[localhost] due to
exception, java.io.IOException: GC overhead limit exceeded |
org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ Journal
Checkpoint Worker
java.io.IOException: *GC overhead limit exceeded*
    at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40)[activemq-client-5.15.9.jar:5.15.9]
    at
org.apache.activemq.store.kahadb.MessageDatabase$CheckpointRunner.run(MessageDatabase.java:451)[activemq-kahadb-store-5.15.9.jar:5.15.9]
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)[:1.8.0_301]
    at java.util.concurrent.FutureTask.runAndReset(Unknown
Source)[:1.8.0_301]
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)[:1.8.0_301]
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)[:1.8.0_301]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)[:1.8.0_301]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)[:1.8.0_301]
    at java.lang.Thread.run(Unknown Source)[:1.8.0_301]

It seems to me that the mKahadb is using a lot more memory ram to handle
the queues and topics than the default kahadb.
Despite that, I realized that maybe the Java is not configured to use the
max possible Memory Ram that the customer has available for the ActiveMQ.

So, I set up the *JAVA_OPTS *system env in our customer with the values
 -Xms512m -Xmx8196m. After that, I restarted the ActiveMQ but I still
receive the "*GC overhead limit exceeded" *exception*. *I don't know if I
need to do more config changes.

Questions:
1. Is the memory ram used by the Java GC a known problem when we set up the
ActiveMQ to use the mKahadb?
2. How do I know how much memory ram the ActiveMQ is currently using for
the GC purpose?

Note: The devices that send messages to our ActiveMQ broker run in a poor
network, so they can become offline for hours.

Thanks
*Rafael Pimenta*


Em qua., 5 de out. de 2022 às 10:18, Rafael Pimenta <ra...@gmail.com>
escreveu:

> As suggested by Mark Johnson I'm evaluating a multi-kahadb setup:
>         <persistenceAdapter>
>         <mKahaDB directory="${activemq.base}/data/kahadb">
>             <filteredPersistenceAdapters>
>             <!-- kahaDB per destinations -->
>             <filteredKahaDB perDestination="true">
>                 <persistenceAdapter>
>                 <kahaDB journalMaxFileLength="1mb"/>
>                 </persistenceAdapter>
>             </filteredKahaDB>
>             </filteredPersistenceAdapters>
>         </mKahaDB>
>         </persistenceAdapter>
>
> But, after that, I'm facing the following warning:
> 2022-10-04 18:05:23,555 | WARN  | Destination name is longer than
> 'MaximumFileNameLength' system property, potential problem with recovery
> can result from name truncation. |
> org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter | ActiveMQ
> Transport: tcp:///192.168.100.123:34926@1883
>
> Where can I change the MaximumFileNameLength to be greater than 64
> characters?
> I saw this ticket related to the underlying problem, but I can't see where
> to change that configuration:
> https://issues.apache.org/jira/browse/AMQ-4183
>
> Att,
> *Rafael Pimenta*
>
> Em seg., 3 de out. de 2022 às 10:34, Rafael Pimenta <
> rafa.spimenta@gmail.com> escreveu:
>
>> Hello Marco,
>> First of All, Thanks for your reply and detailed explanation.
>>
>> To avoid increasing the KahaDB with DLQ queues, we disable the DLQ, so
>> expired messages will be discarded automatically:
>>
>> <deadLetterStrategy>
>>   <sharedDeadLetterStrategy processExpired="false" />
>> </deadLetterStrategy>
>>
>> Because of that configuration, I assume that the problems fall on durable
>> topics.
>> As we know the number of durable topics must be created every time our
>> system start, I'll study virtual topics and check if it is suitable for our
>> purposes. I'll let you ASAP.
>>
>> Thanks!
>> *Rafael Pimenta*
>>
>>
>>
>>
>> Em dom., 2 de out. de 2022 às 08:08, Marco Bettiol <ja...@gmail.com>
>> escreveu:
>>
>>> Hi Rafael,
>>> I’m just an ActiveMQ classic user.
>>>
>>> I will report the behavior of ActiveMQ I experienced.
>>>
>>> As far as I know the ActiveMQ classic journal is composed of append only
>>> files.
>>> Each file should be by default 32MB in size (this can be changed, I think
>>> is controlled by journalLogFileSize)
>>> A journal file gets released only when ALL the messages within are
>>> consumed.
>>> Furthermore, messages form different queues are appended to and mixed
>>> within the same file until the maximum size gets reached.
>>>
>>> As consequence if you have unprocessed messages (DLQ or offline
>>> consumers)
>>> the used storage will increase, usually in the same order of magnitude as
>>> the total traffic served by the broker: a small percentage of unconsumed
>>> messages will prevent journal file cleanup.
>>>
>>> For example an unprocessed 1KByte message will retain 32MB on disk.
>>>
>>> As far as I know there is no building functionality to perform journal
>>> compaction.
>>>
>>> In our case the developed internally a small application to offload
>>> unprocessed messages to an external database/ requeue DLQ messages to the
>>> same DLQ in order to free storage.
>>>
>>> In my experience persistent Topics are quite hard to monitor, if the list
>>> of subscribers is not dynamic  (known when application starts) you could
>>> think to use VirtualTopics so that you will get back to a normal queue
>>> uses
>>> case for each consumer.
>>>
>>> Regards,
>>> Marc
>>>
>>> Il giorno gio 29 set 2022 alle ore 23:06 Rafael Pimenta <
>>> rafa.spimenta@gmail.com> ha scritto:
>>>
>>> > Hi Dears,
>>> > I'm using ActiveMQ 5.15.9 and after about one month our customers
>>> started
>>> > to complain about the Kahadb folder increase until reaching its limit
>>> and
>>> > crash the system. Then they delete the folder to recover the system,
>>> but we
>>> > are losing data in this process.
>>> > We activate the kahadb log and seems that some old messages are getting
>>> > stuck for some reason and do not allow the ActiveMQ to delete the
>>> db-x.log
>>> > files:
>>> >
>>> > *2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244, 243],
>>> > 245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
>>> > 250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247], 252=[140,
>>> > 108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141],
>>> 256=[256,
>>> > 250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259],
>>> 260=[259,
>>> > 260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262,
>>> 254,
>>> > 263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
>>> > 268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263], 270=[159],
>>> > 271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274,
>>> 271],
>>> > 275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277,
>>> 278],
>>> > 279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282],
>>> 283=[159],
>>> > 284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288,
>>> 287],
>>> > 289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
>>> > 294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297],
>>> 298=[174,
>>> > 159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302],
>>> 303=[302,
>>> > 303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308,
>>> 306,
>>> > 287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311],
>>> 313=[312,
>>> > 313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
>>> > 318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
>>> > 323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326,
>>> 278,
>>> > 294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329,
>>> 330],
>>> > 331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251,
>>> 333,
>>> > 253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337,
>>> 339],
>>> > 342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
>>> > 346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347],
>>> 349=[348,
>>> > 337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302, 351,
>>> > 271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21],
>>> 355=[353,
>>> > 355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356,
>>> 358],
>>> > 359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
>>> > 266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361,
>>> 362],
>>> > 363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
>>> > 367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367], 370=[369,
>>> > 370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372,
>>> 373],
>>> > 374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376,
>>> 377],
>>> > 378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
>>> > 381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383],
>>> 385=[384,
>>> > 385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337, 385,
>>> > 386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384, 388,
>>> > 389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
>>> > 392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
>>> > 396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337,
>>> 274,
>>> > 243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34],
>>> 399=[380,
>>> > 397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
>>> > 401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37, 38],
>>> > 406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407],
>>> 410=[409,
>>> > 410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401, 409,
>>> > 389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
>>> > 419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421],
>>> 422=[385,
>>> > 419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394, 414],
>>> > 424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422, 425,
>>> > 426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400,
>>> 424,
>>> > 403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414, 431],
>>> > 432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435],
>>> 436=[49,
>>> > 51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52,
>>> 40],
>>> > 440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
>>> > 444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446],
>>> 447=[438,
>>> > 446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449], 450=[53,
>>> > 54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394, 414],
>>> > 453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
>>> > 457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459],
>>> 460=[49, 53,
>>> > 57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462],
>>> 463=[58,
>>> > 60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466,
>>> 467],
>>> > 468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470,
>>> 471],
>>> > 472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474,
>>> 475],
>>> > 476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
>>> > 480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483,
>>> 484],
>>> > 485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64,
>>> 66],
>>> > 490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493], 494=[493,
>>> > 494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
>>> > 498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
>>> > 501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490,
>>> 459,
>>> > 446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
>>> > 508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511],
>>> 512=[512,
>>> > 438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76],
>>> 517=[515,
>>> > 517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500,
>>> 438,
>>> > 503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511], 520=[519,
>>> > 520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
>>> > 524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525,
>>> 526],
>>> > 527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529, 531],
>>> > 532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517,
>>> 534,
>>> > 535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71],
>>> 540=[538,
>>> > 540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544,
>>> 543],
>>> > 545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531, 548,
>>> > 535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
>>> > 552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554], 555=[97],
>>> > 556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558],
>>> 559=[64, 68,
>>> > 70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70, 55,
>>> > 105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
>>> > 566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568,
>>> 569],
>>> > 570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
>>> > 574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577,
>>> 561,
>>> > 554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
>>> > 581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
>>> > 585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588],
>>> 589=[211],
>>> > 590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
>>> > 594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596,
>>> 597],
>>> > 598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597,
>>> 599,
>>> > 600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602,
>>> 586,
>>> > 604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
>>> > 608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609,
>>> 611,
>>> > 586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599,
>>> 600],
>>> > 614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615, 599,
>>> > 616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618],
>>> 619=[618,
>>> > 619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621,
>>> 622],
>>> > 623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625, 616,
>>> > 621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615,
>>> 619,
>>> > 622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
>>> > 633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631, 634,
>>> > 635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622],
>>> 639=[243]} |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
>>> > contained ack(s) refer to referenced file: [244, 243] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
>>> > contained ack(s) refer to referenced file: [123] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
>>> > contained ack(s) refer to referenced file: [244, 246] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
>>> > contained ack(s) refer to referenced file: [246, 247] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,564 | WARN  | *Stealing link for clientId
>>> > TR_VP-018:30246 From Connection Transport Connection to: tcp://
>>> > 172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker |
>>> > ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
>>> > contained ack(s) refer to referenced file: [123] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
>>> > contained ack(s) refer to referenced file: [249, 247] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
>>> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
>>> 237] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
>>> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
>>> > contained ack(s) refer to referenced file: [123] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
>>> > contained ack(s) refer to referenced file: [249, 247] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
>>> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
>>> 237] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
>>> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
>>> > contained ack(s) refer to referenced file: [140, 108, 97] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
>>> > contained ack(s) refer to referenced file: [249, 253, 251] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
>>> > contained ack(s) refer to referenced file: [253, 254] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
>>> > contained ack(s) refer to referenced file: [140, 141] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker not removing data file: 252 as contained ack(s)
>>> refer to
>>> > referenced file: [140, 108, 97] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
>>> > contained ack(s) refer to referenced file: [249, 253, 251] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
>>> > contained ack(s) refer to referenced file: [253, 254] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
>>> > contained ack(s) refer to referenced file: [140, 141] |
>>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>>> > Checkpoint Worker
>>> >
>>> > We also use durable topics as the devices that subscribe to these
>>> topics
>>> > and send MQTT messages to others work in a very bad network
>>> environment,
>>> > this is the way we found to avoid data loss while offline.
>>> > I think the problem is related to durable topics with offline
>>> > subscriptions, but how can I confirm this assumption, eg which topics?
>>> What
>>> > signatures?
>>> >
>>> > I much appreciate any thoughts!
>>> > Thanks in advance
>>> >
>>>
>>

Re: Kahadb log files increase until reaches its limit

[Rafael Pimenta <ra...@gmail.com>]

As suggested by Mark Johnson I'm evaluating a multi-kahadb setup:
        <persistenceAdapter>
        <mKahaDB directory="${activemq.base}/data/kahadb">
            <filteredPersistenceAdapters>
            <!-- kahaDB per destinations -->
            <filteredKahaDB perDestination="true">
                <persistenceAdapter>
                <kahaDB journalMaxFileLength="1mb"/>
                </persistenceAdapter>
            </filteredKahaDB>
            </filteredPersistenceAdapters>
        </mKahaDB>
        </persistenceAdapter>

But, after that, I'm facing the following warning:
2022-10-04 18:05:23,555 | WARN  | Destination name is longer than
'MaximumFileNameLength' system property, potential problem with recovery
can result from name truncation. |
org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter | ActiveMQ
Transport: tcp:///192.168.100.123:34926@1883

Where can I change the MaximumFileNameLength to be greater than 64
characters?
I saw this ticket related to the underlying problem, but I can't see where
to change that configuration: https://issues.apache.org/jira/browse/AMQ-4183

Att,
*Rafael Pimenta*

Em seg., 3 de out. de 2022 às 10:34, Rafael Pimenta <ra...@gmail.com>
escreveu:

> Hello Marco,
> First of All, Thanks for your reply and detailed explanation.
>
> To avoid increasing the KahaDB with DLQ queues, we disable the DLQ, so
> expired messages will be discarded automatically:
>
> <deadLetterStrategy>
>   <sharedDeadLetterStrategy processExpired="false" />
> </deadLetterStrategy>
>
> Because of that configuration, I assume that the problems fall on durable
> topics.
> As we know the number of durable topics must be created every time our
> system start, I'll study virtual topics and check if it is suitable for our
> purposes. I'll let you ASAP.
>
> Thanks!
> *Rafael Pimenta*
>
>
>
>
> Em dom., 2 de out. de 2022 às 08:08, Marco Bettiol <ja...@gmail.com>
> escreveu:
>
>> Hi Rafael,
>> I’m just an ActiveMQ classic user.
>>
>> I will report the behavior of ActiveMQ I experienced.
>>
>> As far as I know the ActiveMQ classic journal is composed of append only
>> files.
>> Each file should be by default 32MB in size (this can be changed, I think
>> is controlled by journalLogFileSize)
>> A journal file gets released only when ALL the messages within are
>> consumed.
>> Furthermore, messages form different queues are appended to and mixed
>> within the same file until the maximum size gets reached.
>>
>> As consequence if you have unprocessed messages (DLQ or offline consumers)
>> the used storage will increase, usually in the same order of magnitude as
>> the total traffic served by the broker: a small percentage of unconsumed
>> messages will prevent journal file cleanup.
>>
>> For example an unprocessed 1KByte message will retain 32MB on disk.
>>
>> As far as I know there is no building functionality to perform journal
>> compaction.
>>
>> In our case the developed internally a small application to offload
>> unprocessed messages to an external database/ requeue DLQ messages to the
>> same DLQ in order to free storage.
>>
>> In my experience persistent Topics are quite hard to monitor, if the list
>> of subscribers is not dynamic  (known when application starts) you could
>> think to use VirtualTopics so that you will get back to a normal queue
>> uses
>> case for each consumer.
>>
>> Regards,
>> Marc
>>
>> Il giorno gio 29 set 2022 alle ore 23:06 Rafael Pimenta <
>> rafa.spimenta@gmail.com> ha scritto:
>>
>> > Hi Dears,
>> > I'm using ActiveMQ 5.15.9 and after about one month our customers
>> started
>> > to complain about the Kahadb folder increase until reaching its limit
>> and
>> > crash the system. Then they delete the folder to recover the system,
>> but we
>> > are losing data in this process.
>> > We activate the kahadb log and seems that some old messages are getting
>> > stuck for some reason and do not allow the ActiveMQ to delete the
>> db-x.log
>> > files:
>> >
>> > *2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244, 243],
>> > 245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
>> > 250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247], 252=[140,
>> > 108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141], 256=[256,
>> > 250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259],
>> 260=[259,
>> > 260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262,
>> 254,
>> > 263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
>> > 268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263], 270=[159],
>> > 271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274,
>> 271],
>> > 275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277,
>> 278],
>> > 279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282],
>> 283=[159],
>> > 284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288,
>> 287],
>> > 289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
>> > 294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297],
>> 298=[174,
>> > 159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302],
>> 303=[302,
>> > 303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308,
>> 306,
>> > 287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311],
>> 313=[312,
>> > 313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
>> > 318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
>> > 323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326, 278,
>> > 294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329,
>> 330],
>> > 331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251,
>> 333,
>> > 253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337,
>> 339],
>> > 342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
>> > 346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347], 349=[348,
>> > 337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302, 351,
>> > 271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21], 355=[353,
>> > 355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356, 358],
>> > 359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
>> > 266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361, 362],
>> > 363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
>> > 367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367], 370=[369,
>> > 370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372, 373],
>> > 374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376,
>> 377],
>> > 378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
>> > 381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383], 385=[384,
>> > 385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337, 385,
>> > 386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384, 388,
>> > 389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
>> > 392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
>> > 396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337,
>> 274,
>> > 243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34],
>> 399=[380,
>> > 397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
>> > 401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37, 38],
>> > 406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407],
>> 410=[409,
>> > 410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401, 409,
>> > 389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
>> > 419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421], 422=[385,
>> > 419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394, 414],
>> > 424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422, 425,
>> > 426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400, 424,
>> > 403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414, 431],
>> > 432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435],
>> 436=[49,
>> > 51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52, 40],
>> > 440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
>> > 444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446], 447=[438,
>> > 446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449], 450=[53,
>> > 54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394, 414],
>> > 453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
>> > 457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459], 460=[49,
>> 53,
>> > 57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462],
>> 463=[58,
>> > 60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466,
>> 467],
>> > 468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470,
>> 471],
>> > 472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474,
>> 475],
>> > 476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
>> > 480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483,
>> 484],
>> > 485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64,
>> 66],
>> > 490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493], 494=[493,
>> > 494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
>> > 498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
>> > 501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490,
>> 459,
>> > 446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
>> > 508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511],
>> 512=[512,
>> > 438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76], 517=[515,
>> > 517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500,
>> 438,
>> > 503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511], 520=[519,
>> > 520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
>> > 524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525,
>> 526],
>> > 527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529, 531],
>> > 532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517, 534,
>> > 535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71],
>> 540=[538,
>> > 540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544, 543],
>> > 545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531, 548,
>> > 535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
>> > 552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554], 555=[97],
>> > 556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558], 559=[64,
>> 68,
>> > 70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70, 55,
>> > 105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
>> > 566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568,
>> 569],
>> > 570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
>> > 574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577, 561,
>> > 554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
>> > 581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
>> > 585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588],
>> 589=[211],
>> > 590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
>> > 594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596,
>> 597],
>> > 598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597, 599,
>> > 600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602, 586,
>> > 604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
>> > 608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609, 611,
>> > 586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599, 600],
>> > 614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615, 599,
>> > 616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618], 619=[618,
>> > 619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621,
>> 622],
>> > 623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625, 616,
>> > 621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615, 619,
>> > 622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
>> > 633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631, 634,
>> > 635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622],
>> 639=[243]} |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
>> > contained ack(s) refer to referenced file: [244, 243] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
>> > contained ack(s) refer to referenced file: [123] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
>> > contained ack(s) refer to referenced file: [244, 246] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
>> > contained ack(s) refer to referenced file: [246, 247] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,564 | WARN  | *Stealing link for clientId
>> > TR_VP-018:30246 From Connection Transport Connection to: tcp://
>> > 172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker |
>> > ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
>> > contained ack(s) refer to referenced file: [123] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
>> > contained ack(s) refer to referenced file: [249, 247] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
>> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
>> 237] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
>> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
>> > contained ack(s) refer to referenced file: [123] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
>> > contained ack(s) refer to referenced file: [249, 247] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
>> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
>> 237] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
>> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
>> > contained ack(s) refer to referenced file: [140, 108, 97] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
>> > contained ack(s) refer to referenced file: [249, 253, 251] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
>> > contained ack(s) refer to referenced file: [253, 254] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
>> > contained ack(s) refer to referenced file: [140, 141] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker not removing data file: 252 as contained ack(s) refer
>> to
>> > referenced file: [140, 108, 97] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
>> > contained ack(s) refer to referenced file: [249, 253, 251] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
>> > contained ack(s) refer to referenced file: [253, 254] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
>> > contained ack(s) refer to referenced file: [140, 141] |
>> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
>> > Checkpoint Worker
>> >
>> > We also use durable topics as the devices that subscribe to these topics
>> > and send MQTT messages to others work in a very bad network environment,
>> > this is the way we found to avoid data loss while offline.
>> > I think the problem is related to durable topics with offline
>> > subscriptions, but how can I confirm this assumption, eg which topics?
>> What
>> > signatures?
>> >
>> > I much appreciate any thoughts!
>> > Thanks in advance
>> >
>>
>

Re: Kahadb log files increase until reaches its limit

[Rafael Pimenta <ra...@gmail.com>]

Hello Marco,
First of All, Thanks for your reply and detailed explanation.

To avoid increasing the KahaDB with DLQ queues, we disable the DLQ, so
expired messages will be discarded automatically:

<deadLetterStrategy>
  <sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>

Because of that configuration, I assume that the problems fall on durable
topics.
As we know the number of durable topics must be created every time our
system start, I'll study virtual topics and check if it is suitable for our
purposes. I'll let you ASAP.

Thanks!
*Rafael Pimenta*




Em dom., 2 de out. de 2022 às 08:08, Marco Bettiol <ja...@gmail.com>
escreveu:

> Hi Rafael,
> I’m just an ActiveMQ classic user.
>
> I will report the behavior of ActiveMQ I experienced.
>
> As far as I know the ActiveMQ classic journal is composed of append only
> files.
> Each file should be by default 32MB in size (this can be changed, I think
> is controlled by journalLogFileSize)
> A journal file gets released only when ALL the messages within are
> consumed.
> Furthermore, messages form different queues are appended to and mixed
> within the same file until the maximum size gets reached.
>
> As consequence if you have unprocessed messages (DLQ or offline consumers)
> the used storage will increase, usually in the same order of magnitude as
> the total traffic served by the broker: a small percentage of unconsumed
> messages will prevent journal file cleanup.
>
> For example an unprocessed 1KByte message will retain 32MB on disk.
>
> As far as I know there is no building functionality to perform journal
> compaction.
>
> In our case the developed internally a small application to offload
> unprocessed messages to an external database/ requeue DLQ messages to the
> same DLQ in order to free storage.
>
> In my experience persistent Topics are quite hard to monitor, if the list
> of subscribers is not dynamic  (known when application starts) you could
> think to use VirtualTopics so that you will get back to a normal queue uses
> case for each consumer.
>
> Regards,
> Marc
>
> Il giorno gio 29 set 2022 alle ore 23:06 Rafael Pimenta <
> rafa.spimenta@gmail.com> ha scritto:
>
> > Hi Dears,
> > I'm using ActiveMQ 5.15.9 and after about one month our customers started
> > to complain about the Kahadb folder increase until reaching its limit and
> > crash the system. Then they delete the folder to recover the system, but
> we
> > are losing data in this process.
> > We activate the kahadb log and seems that some old messages are getting
> > stuck for some reason and do not allow the ActiveMQ to delete the
> db-x.log
> > files:
> >
> > *2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244, 243],
> > 245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
> > 250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247], 252=[140,
> > 108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141], 256=[256,
> > 250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259], 260=[259,
> > 260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262, 254,
> > 263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
> > 268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263], 270=[159],
> > 271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274, 271],
> > 275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277, 278],
> > 279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282],
> 283=[159],
> > 284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288,
> 287],
> > 289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
> > 294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297], 298=[174,
> > 159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302],
> 303=[302,
> > 303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308,
> 306,
> > 287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311],
> 313=[312,
> > 313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
> > 318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
> > 323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326, 278,
> > 294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329,
> 330],
> > 331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251, 333,
> > 253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337, 339],
> > 342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
> > 346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347], 349=[348,
> > 337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302, 351,
> > 271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21], 355=[353,
> > 355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356, 358],
> > 359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
> > 266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361, 362],
> > 363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
> > 367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367], 370=[369,
> > 370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372, 373],
> > 374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376, 377],
> > 378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
> > 381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383], 385=[384,
> > 385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337, 385,
> > 386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384, 388,
> > 389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
> > 392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
> > 396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337,
> 274,
> > 243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34],
> 399=[380,
> > 397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
> > 401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37, 38],
> > 406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407],
> 410=[409,
> > 410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401, 409,
> > 389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
> > 419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421], 422=[385,
> > 419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394, 414],
> > 424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422, 425,
> > 426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400, 424,
> > 403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414, 431],
> > 432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435],
> 436=[49,
> > 51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52, 40],
> > 440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
> > 444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446], 447=[438,
> > 446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449], 450=[53,
> > 54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394, 414],
> > 453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
> > 457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459], 460=[49,
> 53,
> > 57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462],
> 463=[58,
> > 60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466, 467],
> > 468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470, 471],
> > 472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474, 475],
> > 476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
> > 480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483, 484],
> > 485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64,
> 66],
> > 490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493], 494=[493,
> > 494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
> > 498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
> > 501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490,
> 459,
> > 446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
> > 508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511], 512=[512,
> > 438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76], 517=[515,
> > 517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500,
> 438,
> > 503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511], 520=[519,
> > 520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
> > 524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525,
> 526],
> > 527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529, 531],
> > 532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517, 534,
> > 535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71],
> 540=[538,
> > 540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544, 543],
> > 545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531, 548,
> > 535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
> > 552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554], 555=[97],
> > 556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558], 559=[64,
> 68,
> > 70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70, 55,
> > 105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
> > 566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568, 569],
> > 570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
> > 574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577, 561,
> > 554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
> > 581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
> > 585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588],
> 589=[211],
> > 590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
> > 594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596, 597],
> > 598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597, 599,
> > 600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602, 586,
> > 604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
> > 608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609, 611,
> > 586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599, 600],
> > 614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615, 599,
> > 616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618], 619=[618,
> > 619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621, 622],
> > 623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625, 616,
> > 621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615, 619,
> > 622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
> > 633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631, 634,
> > 635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622],
> 639=[243]} |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
> > contained ack(s) refer to referenced file: [244, 243] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
> > contained ack(s) refer to referenced file: [123] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
> > contained ack(s) refer to referenced file: [244, 246] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
> > contained ack(s) refer to referenced file: [246, 247] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,564 | WARN  | *Stealing link for clientId
> > TR_VP-018:30246 From Connection Transport Connection to: tcp://
> > 172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker |
> > ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
> > contained ack(s) refer to referenced file: [123] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
> > contained ack(s) refer to referenced file: [249, 247] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
> 237] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
> > contained ack(s) refer to referenced file: [123] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
> > contained ack(s) refer to referenced file: [249, 247] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
> > contained ack(s) refer to referenced file: [163, 211, 249, 250, 234,
> 237] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
> > contained ack(s) refer to referenced file: [250, 243, 251, 247] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
> > contained ack(s) refer to referenced file: [140, 108, 97] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
> > contained ack(s) refer to referenced file: [249, 253, 251] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
> > contained ack(s) refer to referenced file: [253, 254] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
> > contained ack(s) refer to referenced file: [140, 141] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker not removing data file: 252 as contained ack(s) refer
> to
> > referenced file: [140, 108, 97] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
> > contained ack(s) refer to referenced file: [249, 253, 251] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
> > contained ack(s) refer to referenced file: [253, 254] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> > *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
> > contained ack(s) refer to referenced file: [140, 141] |
> > org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> > Checkpoint Worker
> >
> > We also use durable topics as the devices that subscribe to these topics
> > and send MQTT messages to others work in a very bad network environment,
> > this is the way we found to avoid data loss while offline.
> > I think the problem is related to durable topics with offline
> > subscriptions, but how can I confirm this assumption, eg which topics?
> What
> > signatures?
> >
> > I much appreciate any thoughts!
> > Thanks in advance
> >
>

Re: Kahadb log files increase until reaches its limit

[Marco Bettiol <ja...@gmail.com>]

Hi Rafael,
I’m just an ActiveMQ classic user.

I will report the behavior of ActiveMQ I experienced.

As far as I know the ActiveMQ classic journal is composed of append only
files.
Each file should be by default 32MB in size (this can be changed, I think
is controlled by journalLogFileSize)
A journal file gets released only when ALL the messages within are consumed.
Furthermore, messages form different queues are appended to and mixed
within the same file until the maximum size gets reached.

As consequence if you have unprocessed messages (DLQ or offline consumers)
the used storage will increase, usually in the same order of magnitude as
the total traffic served by the broker: a small percentage of unconsumed
messages will prevent journal file cleanup.

For example an unprocessed 1KByte message will retain 32MB on disk.

As far as I know there is no building functionality to perform journal
compaction.

In our case the developed internally a small application to offload
unprocessed messages to an external database/ requeue DLQ messages to the
same DLQ in order to free storage.

In my experience persistent Topics are quite hard to monitor, if the list
of subscribers is not dynamic  (known when application starts) you could
think to use VirtualTopics so that you will get back to a normal queue uses
case for each consumer.

Regards,
Marc

Il giorno gio 29 set 2022 alle ore 23:06 Rafael Pimenta <
rafa.spimenta@gmail.com> ha scritto:

> Hi Dears,
> I'm using ActiveMQ 5.15.9 and after about one month our customers started
> to complain about the Kahadb folder increase until reaching its limit and
> crash the system. Then they delete the folder to recover the system, but we
> are losing data in this process.
> We activate the kahadb log and seems that some old messages are getting
> stuck for some reason and do not allow the ActiveMQ to delete the db-x.log
> files:
>
> *2022-09-29 17:39:51,555 | TRACE | ackMessageFileMap:* {244=[244, 243],
> 245=[123], 246=[244, 246], 247=[246, 247], 248=[123], 249=[249, 247],
> 250=[163, 211, 249, 250, 234, 237], 251=[250, 243, 251, 247], 252=[140,
> 108, 97], 253=[249, 253, 251], 254=[253, 254], 255=[140, 141], 256=[256,
> 250, 251, 253, 254], 257=[256, 257], 258=[141], 259=[257, 259], 260=[259,
> 260, 254], 261=[140], 262=[260, 262], 263=[262, 263], 264=[264, 262, 254,
> 263], 265=[264, 265, 253], 266=[265, 249, 266, 250, 253], 267=[159],
> 268=[266, 268, 253, 79], 269=[259, 268, 269, 262, 254, 263], 270=[159],
> 271=[269, 271], 272=[272, 265, 266, 271], 273=[163], 274=[272, 274, 271],
> 275=[274, 275], 276=[163], 277=[265, 274, 266, 275, 277], 278=[277, 278],
> 279=[163], 280=[280, 278, 266], 281=[280, 281], 282=[281, 282], 283=[159],
> 284=[284, 282], 285=[284, 285], 286=[174], 287=[285, 287], 288=[288, 287],
> 289=[174], 290=[288, 290], 291=[290, 291], 292=[163], 293=[293, 291],
> 294=[293, 294], 295=[294, 295], 296=[296, 295], 297=[296, 297], 298=[174,
> 159], 299=[297, 299], 300=[300, 299], 301=[180], 302=[300, 302], 303=[302,
> 303], 304=[180], 305=[305, 303], 306=[305, 306], 307=[182], 308=[308, 306,
> 287], 309=[308, 309], 310=[182], 311=[309, 311], 312=[312, 311], 313=[312,
> 313], 314=[313, 314], 315=[284, 314, 315], 316=[174], 317=[317, 315],
> 318=[317, 318], 319=[198], 320=[320, 318], 321=[320, 321], 322=[199],
> 323=[321, 323], 324=[324, 323], 325=[199], 326=[272, 274, 324, 326, 278,
> 294, 296], 327=[326, 327], 328=[328, 327], 329=[328, 329], 330=[329, 330],
> 331=[208], 332=[332, 330], 333=[332, 333], 334=[208], 335=[250, 251, 333,
> 253, 335], 336=[336, 335], 337=[336, 337, 211], 338=[20], 339=[337, 339],
> 342=[337, 342, 339], 343=[342, 343], 344=[20, 21], 345=[345, 343],
> 346=[345, 346], 347=[337, 345, 346, 347], 348=[348, 345, 347], 349=[348,
> 337, 349], 350=[349, 350], 351=[272, 337, 274, 294, 296, 350, 302, 351,
> 271], 352=[352, 337, 351], 353=[352, 353, 337, 285], 354=[21], 355=[353,
> 355], 356=[337, 274, 355, 356, 294, 299, 302], 357=[24], 358=[356, 358],
> 359=[358, 359], 360=[256, 272, 337, 274, 294, 359, 263, 360, 296, 265,
> 266], 361=[337, 274, 263, 359, 360, 361, 265, 253, 302], 362=[361, 362],
> 363=[360, 362, 363], 364=[364, 361, 363], 365=[24], 366=[364, 366],
> 367=[337, 345, 346, 366, 367, 351], 368=[21], 369=[369, 367], 370=[369,
> 370], 371=[28], 372=[337, 369, 370, 372, 345, 346, 351], 373=[372, 373],
> 374=[373, 374], 375=[374, 375], 376=[376, 345, 375, 351], 377=[376, 377],
> 378=[369, 374, 376, 377, 345, 378, 351], 379=[28], 380=[380, 378],
> 381=[380, 381], 382=[28], 383=[381, 383], 384=[384, 381, 383], 385=[384,
> 385, 366], 386=[352, 384, 385, 386, 350, 351, 367, 383], 387=[337, 385,
> 386, 387, 345, 378, 348, 366, 351], 388=[388, 387], 389=[352, 384, 388,
> 389, 351, 383], 390=[376, 369, 380, 389, 390, 374], 391=[390, 391],
> 392=[34], 393=[393, 387, 389, 390, 391], 394=[393, 394], 395=[34],
> 396=[385, 259, 387, 390, 296, 360, 265, 394, 266, 363, 396, 366, 337, 274,
> 243, 277, 278, 250, 284, 253], 397=[396, 397, 394], 398=[28, 34], 399=[380,
> 397, 374, 399], 400=[400, 384, 390, 362, 380, 396, 350, 399, 351],
> 401=[400, 401], 402=[37], 403=[401, 403], 404=[404, 403], 405=[37, 38],
> 406=[404, 406], 407=[406, 407], 408=[34, 38, 39], 409=[409, 407], 410=[409,
> 410], 411=[409, 410, 411], 412=[40, 39], 413=[413, 411], 414=[401, 409,
> 389, 413, 414], 415=[40], 416=[416, 414], 417=[416, 417], 418=[38],
> 419=[417, 419], 420=[420, 414, 419], 421=[400, 403, 420, 421], 422=[385,
> 419, 420, 421, 422], 423=[387, 419, 404, 420, 421, 422, 423, 394, 414],
> 424=[424, 419, 420, 423], 425=[424, 425], 426=[419, 404, 420, 422, 425,
> 426, 394, 414], 427=[426, 419, 427, 420], 428=[428, 427], 429=[400, 424,
> 403, 428, 429], 430=[48], 431=[419, 403, 404, 420, 424, 429, 414, 431],
> 432=[432, 431], 433=[48, 49], 434=[432, 434], 435=[424, 434, 435], 436=[49,
> 51], 437=[437, 435], 438=[428, 437, 438, 431], 439=[48, 49, 51, 52, 40],
> 440=[438, 440], 441=[440, 441], 442=[52, 53, 21], 443=[438, 441, 443],
> 444=[438, 440, 443, 444], 445=[438, 444, 445], 446=[445, 446], 447=[438,
> 446, 447], 448=[448, 438, 440, 445, 446, 447], 449=[448, 449], 450=[53,
> 54], 451=[449, 451], 452=[337, 451, 419, 452, 404, 420, 438, 394, 414],
> 453=[49, 54, 55, 40], 454=[452, 454], 455=[454, 455], 456=[55, 57],
> 457=[449, 438, 455, 457, 441], 458=[457, 458], 459=[458, 459], 460=[49, 53,
> 57, 58], 461=[437, 438, 459, 461], 462=[438, 458, 459, 461, 462], 463=[58,
> 60], 464=[464, 462], 465=[464, 465, 459], 466=[465, 466], 467=[466, 467],
> 468=[466, 467, 468], 469=[468, 469, 459], 470=[469, 470], 471=[470, 471],
> 472=[471, 472], 473=[472, 473], 474=[471, 472, 473, 474], 475=[474, 475],
> 476=[475, 476], 477=[476, 477], 478=[477, 478], 479=[34, 40, 60, 61],
> 480=[480, 478], 481=[480, 481], 482=[61], 483=[481, 483], 484=[483, 484],
> 485=[64, 40], 486=[484, 486], 487=[486, 487], 488=[487, 488], 489=[64, 66],
> 490=[488, 490], 491=[490, 491], 492=[66, 67], 493=[491, 493], 494=[493,
> 494], 495=[64, 67, 68], 496=[496, 464, 465, 494], 497=[496, 497, 438],
> 498=[497, 498], 499=[68, 70], 500=[465, 497, 498, 500, 438], 501=[500,
> 501], 502=[70, 71], 503=[420, 484, 501, 437, 438, 486, 503, 457, 490, 459,
> 446], 504=[503, 504], 505=[71, 73], 506=[504, 506], 507=[506, 507],
> 508=[507, 508], 509=[508, 509], 510=[509, 510], 511=[510, 511], 512=[512,
> 438, 511], 513=[73], 514=[512, 514], 515=[514, 515], 516=[76], 517=[515,
> 517], 518=[386, 387, 452, 517, 421, 518, 424, 459, 397, 429, 465, 500, 438,
> 503, 506], 519=[352, 337, 437, 518, 438, 519, 345, 381, 511], 520=[519,
> 520], 521=[519, 520, 521], 522=[521, 522], 523=[515, 521, 522, 523],
> 524=[76, 77], 525=[515, 521, 522, 523, 525], 526=[514, 515, 521, 525, 526],
> 527=[77, 79], 528=[528, 526], 529=[528, 529], 530=[79], 531=[529, 531],
> 532=[531, 532, 520], 533=[532, 533], 534=[533, 534], 535=[512, 517, 534,
> 535], 536=[81], 537=[535, 537], 538=[537, 538], 539=[81, 82, 71], 540=[538,
> 540], 541=[540, 541], 542=[82, 83], 543=[535, 541, 543], 544=[544, 543],
> 545=[544, 545], 546=[83], 547=[545, 547], 548=[544, 545, 547, 531, 548,
> 535, 521, 538, 543], 549=[79], 550=[548, 550], 551=[548, 550, 551],
> 552=[97], 553=[551, 553], 554=[337, 531, 438, 535, 553, 554], 555=[97],
> 556=[548, 554, 538, 556], 557=[545, 556, 557], 558=[557, 558], 559=[64, 68,
> 70], 560=[560, 544, 554, 558], 561=[560, 561], 562=[64, 82, 68, 70, 55,
> 105, 57], 563=[561, 563], 564=[563, 564, 537, 558], 565=[564, 565],
> 566=[565, 566, 556, 558], 567=[566, 567], 568=[567, 568], 569=[568, 569],
> 570=[566, 569, 570, 558], 571=[570, 571], 572=[211], 573=[571, 573],
> 574=[573, 574], 575=[211, 214], 576=[576, 574], 577=[576, 544, 577, 561,
> 554, 557], 578=[97, 105], 579=[577, 579, 565, 566], 580=[579, 580],
> 581=[214], 582=[580, 582], 583=[582, 583], 584=[566, 583, 584, 558],
> 585=[108], 586=[566, 584, 586], 587=[108], 588=[583, 586, 588], 589=[211],
> 590=[586, 588, 590], 591=[590, 591], 592=[211], 593=[593, 586, 591],
> 594=[593, 594, 586], 595=[594, 595], 596=[595, 596, 586], 597=[596, 597],
> 598=[111], 599=[596, 597, 599, 586], 600=[597, 599, 600], 601=[597, 599,
> 600, 601], 602=[601, 602, 586, 588], 603=[211], 604=[583, 599, 602, 586,
> 604], 605=[604, 605], 606=[586, 605, 606], 607=[113, 97, 108, 111],
> 608=[608, 596, 599, 586, 606], 609=[608, 609], 610=[113], 611=[609, 611,
> 586, 588], 612=[611, 612, 583, 599, 586], 613=[608, 612, 613, 599, 600],
> 614=[609, 596, 613, 614, 599, 586], 615=[614, 615], 616=[608, 615, 599,
> 616, 600, 586], 617=[615, 616, 617], 618=[611, 613, 617, 618], 619=[618,
> 619], 620=[234], 621=[611, 613, 619, 621], 622=[608, 609, 611, 621, 622],
> 623=[234], 624=[624, 608, 612, 613, 614, 615, 622], 625=[624, 625, 616,
> 621], 626=[234, 237], 627=[625, 627], 628=[624, 625, 627, 628, 615, 619,
> 622], 629=[237], 630=[628, 630], 631=[630, 631], 632=[97, 108, 79],
> 633=[624, 625, 630, 631, 633, 622], 634=[633, 634], 635=[599, 631, 634,
> 635, 621], 636=[97], 637=[635, 637], 638=[619, 637, 638, 622], 639=[243]} |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,557 | TRACE |* not removing data file: 244 as
> contained ack(s) refer to referenced file: [244, 243] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,559 | TRACE |* not removing data file: 245 as
> contained ack(s) refer to referenced file: [123] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,561 | TRACE |* not removing data file: 246 as
> contained ack(s) refer to referenced file: [244, 246] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,563 | TRACE |* not removing data file: 247 as
> contained ack(s) refer to referenced file: [246, 247] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,564 | WARN  | *Stealing link for clientId
> TR_VP-018:30246 From Connection Transport Connection to: tcp://
> 172.24.96.125:42412 | org.apache.activemq.broker.region.RegionBroker |
> ActiveMQ Transport: tcp:///172.24.96.125:42414@1883
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
> contained ack(s) refer to referenced file: [123] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
> contained ack(s) refer to referenced file: [249, 247] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
> contained ack(s) refer to referenced file: [163, 211, 249, 250, 234, 237] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
> contained ack(s) refer to referenced file: [250, 243, 251, 247] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 248 as
> contained ack(s) refer to referenced file: [123] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 249 as
> contained ack(s) refer to referenced file: [249, 247] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 250 as
> contained ack(s) refer to referenced file: [163, 211, 249, 250, 234, 237] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 251 as
> contained ack(s) refer to referenced file: [250, 243, 251, 247] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 252 as
> contained ack(s) refer to referenced file: [140, 108, 97] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 253 as
> contained ack(s) refer to referenced file: [249, 253, 251] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 254 as
> contained ack(s) refer to referenced file: [253, 254] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
> contained ack(s) refer to referenced file: [140, 141] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker not removing data file: 252 as contained ack(s) refer to
> referenced file: [140, 108, 97] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 253 as
> contained ack(s) refer to referenced file: [249, 253, 251] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE | *not removing data file: 254 as
> contained ack(s) refer to referenced file: [253, 254] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> *2022-09-29 17:39:51,565 | TRACE |* not removing data file: 255 as
> contained ack(s) refer to referenced file: [140, 141] |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
>
> We also use durable topics as the devices that subscribe to these topics
> and send MQTT messages to others work in a very bad network environment,
> this is the way we found to avoid data loss while offline.
> I think the problem is related to durable topics with offline
> subscriptions, but how can I confirm this assumption, eg which topics? What
> signatures?
>
> I much appreciate any thoughts!
> Thanks in advance
>