You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Lipika Rout <li...@gmail.com> on 2021/06/03 18:39:43 UTC
Unable to recover data after activemq upgrade
Hello,
We are using activemq embedded with Tomcat v9.0.26. Recently we upgraded
activemq jar from v5.3.0 to 5.16.0 without realizing that messages stopped
processing for a month in one of the nodes. We are trying to recover data
from 2 journal log files. Usually this happens automatically after restart,
but this time this didn't happen.
Below are the statistics for one of the journal files.
Destination statistics:
- Topics: 0.
- Queues: 0.
Commands without destination:
+ CmdType: KAHA_TRACE_COMMAND (Count: 51407, TotalSize: 31.5 MB (33034964),
~AvrgSize: 642 Byte(s) (642), LastBigSize: 2.39 KB (2448), LastSize: 79
Byte(s) (79))
All commands: 51407 (Total size: 31.5 MB (33034964).
-----------------------------------------------------------
Command statistics:
- Topics: 0 (messages: 0, +subscriptions: 0, -subscription: 0).
- Queues: 0 (messages: 0).
- Other messages: 51407.
Commands:
+ CmdType: KAHA_TRACE_COMMAND (Count: 51407, TotalSize: 31.5 MB (33034964),
~AvrgSize: 642 Byte(s) (642), LastBigSize: 2.39 KB (2448), LastSize: 79
Byte(s) (79))
All commands: 51407 (Total size: 31.5 MB (33034964).
Looking for a solution to replay these messages.
Thanks
Lipika Rout
Re: Unable to recover data after activemq upgrade
Posted by Tim Bain <tb...@alumni.duke.edu>.
Lipika (the OP) responded directly to me with the following content:
We didn't clean start activemq during upgrade. So messages were already
there in the queue when we just upgraded the activemq jar version and
restart the Tomcat server without following any of the migration steps. I
tried to recover data by configuring a 5.3 broker with
ignoreMissingJournalfiles="true" checkForCorruptJournalFiles="true"
checksumJournalFiles="true". None of the messages were able to recover by
the broker. My backup data file looks like corrupted, not able to
recover/read messages. Any other way to read those messages.
Can you describe in a bit more detail how the data file seems to be
corrupted? Is that just a reflection of "I tried to import from it and it
didn't work," or do you have a more concrete indication of corruption?
You should be able to run the broker with increased logging in the KahaDB
module and see more info about what the code thinks is wrong.
https://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup
has some details about how to enable that logging.
Also, in
http://activemq.2283324.n4.nabble.com/How-to-access-KahaDB-db-data-in-activeMQ-5-15-0-td4757076.html
Gary Tully referenced a tool to export the messages from a set of KahaDB
files, so you might try that (and increase logging if necessary) to test
the theory that the data files are corrupted.
Tim
On Sat, Jun 5, 2021, 7:16 AM Tim Bain <tb...@alumni.duke.edu> wrote:
> I was referencing the "Pure static networks" section near the bottom of
> https://activemq.apache.org/networks-of-brokers. That ensures that
> messages are propagated off the 5.3 broker and onto the 5.16 broker even if
> no consumer for those destinations is currently connected to the 5.16
> broker.
>
> Also, do you use durable subscriptions on topics? If so, and if those
> durable subscriptions have been re-created on your 5.16 broker, I'm not
> sure what will happen when you network in the 5.3 broker. I'd expect that
> the prior subscription's state wouldn't be honored, but I'm not sure
> whether your subscriber would receive all messages on the topic from the
> 5.3 broker (i.e. ignoring the fact that some messages had already been
> acked) or none of the messages on the topic from the 5.3 broker (i.e.
> treating the topic as though all of those messages had already been acked).
> Hopefully you don't have any durable subscriptions and don't have to worry
> about this. If you do, I encourage you to test this out in a non-production
> environment to be sure you understand the behavior. There might be
> suggestions we can make if needed, depending on the specifics of your use
> case.
>
> Tim
>
>
> On Fri, Jun 4, 2021, 11:06 AM Phil Ruggera <pr...@gmail.com> wrote:
>
>> What does "include all destinations in the 5.3 -> 5.16 direction, to force
>> all messages to be transferred" entail?
>>
>> On Fri, Jun 4, 2021, 5:45 AM Tim Bain <tb...@alumni.duke.edu> wrote:
>>
>> > One option would be to load the 5.3 data file into a new temporary 5.3
>> > broker and configure it to make a network of brokers with your 5.16.0
>> > broker. You'll likely want to statically include all destinations in the
>> > 5.3 -> 5.16 direction, to force all messages to be transferred to the
>> > 5.16.0 broker. Once that's complete, you can tear down the 5.3 broker.
>> >
>> > Tim
>> >
>> > On Thu, Jun 3, 2021, 1:50 PM Lipika Rout <li...@gmail.com> wrote:
>> >
>> > > Hello,
>> > >
>> > > We are using activemq embedded with Tomcat v9.0.26. Recently we
>> upgraded
>> > > activemq jar from v5.3.0 to 5.16.0 without realizing that messages
>> > stopped
>> > > processing for a month in one of the nodes. We are trying to recover
>> data
>> > > from 2 journal log files. Usually this happens automatically after
>> > restart,
>> > > but this time this didn't happen.
>> > >
>> > > Below are the statistics for one of the journal files.
>> > >
>> > > Destination statistics:
>> > > - Topics: 0.
>> > > - Queues: 0.
>> > >
>> > > Commands without destination:
>> > > + CmdType: KAHA_TRACE_COMMAND (Count: 51407, TotalSize: 31.5 MB
>> > (33034964),
>> > > ~AvrgSize: 642 Byte(s) (642), LastBigSize: 2.39 KB (2448), LastSize:
>> 79
>> > > Byte(s) (79))
>> > > All commands: 51407 (Total size: 31.5 MB (33034964).
>> > > -----------------------------------------------------------
>> > > Command statistics:
>> > > - Topics: 0 (messages: 0, +subscriptions: 0, -subscription: 0).
>> > > - Queues: 0 (messages: 0).
>> > > - Other messages: 51407.
>> > >
>> > > Commands:
>> > > + CmdType: KAHA_TRACE_COMMAND (Count: 51407, TotalSize: 31.5 MB
>> > (33034964),
>> > > ~AvrgSize: 642 Byte(s) (642), LastBigSize: 2.39 KB (2448), LastSize:
>> 79
>> > > Byte(s) (79))
>> > > All commands: 51407 (Total size: 31.5 MB (33034964).
>> > >
>> > > Looking for a solution to replay these messages.
>> > >
>> > > Thanks
>> > > Lipika Rout
>> > >
>> >
>>
>
Re: Unable to recover data after activemq upgrade
Posted by Tim Bain <tb...@alumni.duke.edu>.
I was referencing the "Pure static networks" section near the bottom of
https://activemq.apache.org/networks-of-brokers. That ensures that messages
are propagated off the 5.3 broker and onto the 5.16 broker even if no
consumer for those destinations is currently connected to the 5.16 broker.
Also, do you use durable subscriptions on topics? If so, and if those
durable subscriptions have been re-created on your 5.16 broker, I'm not
sure what will happen when you network in the 5.3 broker. I'd expect that
the prior subscription's state wouldn't be honored, but I'm not sure
whether your subscriber would receive all messages on the topic from the
5.3 broker (i.e. ignoring the fact that some messages had already been
acked) or none of the messages on the topic from the 5.3 broker (i.e.
treating the topic as though all of those messages had already been acked).
Hopefully you don't have any durable subscriptions and don't have to worry
about this. If you do, I encourage you to test this out in a non-production
environment to be sure you understand the behavior. There might be
suggestions we can make if needed, depending on the specifics of your use
case.
Tim
On Fri, Jun 4, 2021, 11:06 AM Phil Ruggera <pr...@gmail.com> wrote:
> What does "include all destinations in the 5.3 -> 5.16 direction, to force
> all messages to be transferred" entail?
>
> On Fri, Jun 4, 2021, 5:45 AM Tim Bain <tb...@alumni.duke.edu> wrote:
>
> > One option would be to load the 5.3 data file into a new temporary 5.3
> > broker and configure it to make a network of brokers with your 5.16.0
> > broker. You'll likely want to statically include all destinations in the
> > 5.3 -> 5.16 direction, to force all messages to be transferred to the
> > 5.16.0 broker. Once that's complete, you can tear down the 5.3 broker.
> >
> > Tim
> >
> > On Thu, Jun 3, 2021, 1:50 PM Lipika Rout <li...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > We are using activemq embedded with Tomcat v9.0.26. Recently we
> upgraded
> > > activemq jar from v5.3.0 to 5.16.0 without realizing that messages
> > stopped
> > > processing for a month in one of the nodes. We are trying to recover
> data
> > > from 2 journal log files. Usually this happens automatically after
> > restart,
> > > but this time this didn't happen.
> > >
> > > Below are the statistics for one of the journal files.
> > >
> > > Destination statistics:
> > > - Topics: 0.
> > > - Queues: 0.
> > >
> > > Commands without destination:
> > > + CmdType: KAHA_TRACE_COMMAND (Count: 51407, TotalSize: 31.5 MB
> > (33034964),
> > > ~AvrgSize: 642 Byte(s) (642), LastBigSize: 2.39 KB (2448), LastSize: 79
> > > Byte(s) (79))
> > > All commands: 51407 (Total size: 31.5 MB (33034964).
> > > -----------------------------------------------------------
> > > Command statistics:
> > > - Topics: 0 (messages: 0, +subscriptions: 0, -subscription: 0).
> > > - Queues: 0 (messages: 0).
> > > - Other messages: 51407.
> > >
> > > Commands:
> > > + CmdType: KAHA_TRACE_COMMAND (Count: 51407, TotalSize: 31.5 MB
> > (33034964),
> > > ~AvrgSize: 642 Byte(s) (642), LastBigSize: 2.39 KB (2448), LastSize: 79
> > > Byte(s) (79))
> > > All commands: 51407 (Total size: 31.5 MB (33034964).
> > >
> > > Looking for a solution to replay these messages.
> > >
> > > Thanks
> > > Lipika Rout
> > >
> >
>
Re: Unable to recover data after activemq upgrade
Posted by Phil Ruggera <pr...@gmail.com>.
What does "include all destinations in the 5.3 -> 5.16 direction, to force
all messages to be transferred" entail?
On Fri, Jun 4, 2021, 5:45 AM Tim Bain <tb...@alumni.duke.edu> wrote:
> One option would be to load the 5.3 data file into a new temporary 5.3
> broker and configure it to make a network of brokers with your 5.16.0
> broker. You'll likely want to statically include all destinations in the
> 5.3 -> 5.16 direction, to force all messages to be transferred to the
> 5.16.0 broker. Once that's complete, you can tear down the 5.3 broker.
>
> Tim
>
> On Thu, Jun 3, 2021, 1:50 PM Lipika Rout <li...@gmail.com> wrote:
>
> > Hello,
> >
> > We are using activemq embedded with Tomcat v9.0.26. Recently we upgraded
> > activemq jar from v5.3.0 to 5.16.0 without realizing that messages
> stopped
> > processing for a month in one of the nodes. We are trying to recover data
> > from 2 journal log files. Usually this happens automatically after
> restart,
> > but this time this didn't happen.
> >
> > Below are the statistics for one of the journal files.
> >
> > Destination statistics:
> > - Topics: 0.
> > - Queues: 0.
> >
> > Commands without destination:
> > + CmdType: KAHA_TRACE_COMMAND (Count: 51407, TotalSize: 31.5 MB
> (33034964),
> > ~AvrgSize: 642 Byte(s) (642), LastBigSize: 2.39 KB (2448), LastSize: 79
> > Byte(s) (79))
> > All commands: 51407 (Total size: 31.5 MB (33034964).
> > -----------------------------------------------------------
> > Command statistics:
> > - Topics: 0 (messages: 0, +subscriptions: 0, -subscription: 0).
> > - Queues: 0 (messages: 0).
> > - Other messages: 51407.
> >
> > Commands:
> > + CmdType: KAHA_TRACE_COMMAND (Count: 51407, TotalSize: 31.5 MB
> (33034964),
> > ~AvrgSize: 642 Byte(s) (642), LastBigSize: 2.39 KB (2448), LastSize: 79
> > Byte(s) (79))
> > All commands: 51407 (Total size: 31.5 MB (33034964).
> >
> > Looking for a solution to replay these messages.
> >
> > Thanks
> > Lipika Rout
> >
>
Re: Unable to recover data after activemq upgrade
Posted by Tim Bain <tb...@alumni.duke.edu>.
One option would be to load the 5.3 data file into a new temporary 5.3
broker and configure it to make a network of brokers with your 5.16.0
broker. You'll likely want to statically include all destinations in the
5.3 -> 5.16 direction, to force all messages to be transferred to the
5.16.0 broker. Once that's complete, you can tear down the 5.3 broker.
Tim
On Thu, Jun 3, 2021, 1:50 PM Lipika Rout <li...@gmail.com> wrote:
> Hello,
>
> We are using activemq embedded with Tomcat v9.0.26. Recently we upgraded
> activemq jar from v5.3.0 to 5.16.0 without realizing that messages stopped
> processing for a month in one of the nodes. We are trying to recover data
> from 2 journal log files. Usually this happens automatically after restart,
> but this time this didn't happen.
>
> Below are the statistics for one of the journal files.
>
> Destination statistics:
> - Topics: 0.
> - Queues: 0.
>
> Commands without destination:
> + CmdType: KAHA_TRACE_COMMAND (Count: 51407, TotalSize: 31.5 MB (33034964),
> ~AvrgSize: 642 Byte(s) (642), LastBigSize: 2.39 KB (2448), LastSize: 79
> Byte(s) (79))
> All commands: 51407 (Total size: 31.5 MB (33034964).
> -----------------------------------------------------------
> Command statistics:
> - Topics: 0 (messages: 0, +subscriptions: 0, -subscription: 0).
> - Queues: 0 (messages: 0).
> - Other messages: 51407.
>
> Commands:
> + CmdType: KAHA_TRACE_COMMAND (Count: 51407, TotalSize: 31.5 MB (33034964),
> ~AvrgSize: 642 Byte(s) (642), LastBigSize: 2.39 KB (2448), LastSize: 79
> Byte(s) (79))
> All commands: 51407 (Total size: 31.5 MB (33034964).
>
> Looking for a solution to replay these messages.
>
> Thanks
> Lipika Rout
>