[jira] [Resolved] (AMQ-6152) KahaDB scheduler log files not being deleted

["Timothy Bish (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AMQ-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Timothy Bish resolved AMQ-6152.
-------------------------------
    Resolution: Fixed

> KahaDB scheduler log files not being deleted
> --------------------------------------------
>
>                 Key: AMQ-6152
>                 URL: https://issues.apache.org/jira/browse/AMQ-6152
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: KahaDB
>    Affects Versions: 5.11.1, 5.12.0, 5.13.0
>         Environment: Linux (issue reported on several versions)
> OpenJDK 1.8.0_72
>            Reporter: Ryan Senior
>            Assignee: Timothy Bish
>             Fix For: 5.14.0, 5.13.2
>
>         Attachments: BasicSchedulerTest.java
>
>
> Something is holding onto KahaDB scheduler log files. We have reports of up to 400GB of scheduler log files. I have tried to isolate the issue and create a minimal example (attached). In the troubleshooting I have done, the scheduler GC process is running, it's just deciding not to GC files that it should be. I have also found behavior inconsistent on the log files it does remove. 
> The ran the attached example/test on 5.10.0, 5.11.1, 5.12.0 and 5.13.0. The test schedules 20 messages that are large enough to cause 4 log files to be created. It then consumes all 20 messages. When on 5.10.0, it behaves like I would expect, files 1-3 are GC'd and the 4th (the current log file) is left. On all other versions I've tried it always leaves the first 2 files, and sometimes will GC the 3rd.
> Below is a snippet from the log of the scheduler process and why it's deciding not to GC these files:
> {noformat}
> 2016-02-01 16:32:33,327 DEBUG [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] Job Scheduler Store Checkpoint started.
> 2016-02-01 16:32:33,338 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] Last update: 4:29447367, full gc candidates set: [1, 2, 3, 4]
> 2016-02-01 16:32:33,338 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] gc candidates after reference check: [1, 2, 3]
> 2016-02-01 16:32:33,338 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] A remove in log 1 has an add still in existance.
> 2016-02-01 16:32:33,339 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] A remove in log 2 has an add still in existance.
> 2016-02-01 16:32:33,339 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] A remove in log 3 has an add still in existance.
> 2016-02-01 16:32:33,339 TRACE [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] gc candidates after removals check: []
> 2016-02-01 16:32:33,339 DEBUG [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] Job Scheduler Store Checkpoint complete.
> 2016-02-01 16:32:33,727 INFO  [main] [o.a.a.s.k.s.JobSchedulerStoreImpl] JobSchedulerStore: test-mq/localhost/scheduler stopped.
> {noformat}
> This issue was originally reported in the Open Source PuppetDB project, ticket [here|https://tickets.puppetlabs.com/browse/PDB-1411].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)