You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Roy Cohen <ro...@hotmail.com> on 2022/05/11 22:35:00 UTC
Artemis isn’t starting up
Hi All
We have an urgent situation with our Artemis brokers pair in production which somehow has left 01 without any DLQs and 02 in a state where it’s not starting up.
The one which isn’t starting up is throwing this Exception, unfortunately I haven’t been able to find much online. Any help would be greatly appreciated
2022-05-11 20:35:38,673 ERROR [org.apache.activemq.artemis.core.server] (main) AMQ224000: Failure in initialisation: java.lang.IllegalStateException: Cannot find queue with id 80
at org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394) [artemis-jms-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65) [artemis-cli-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83) [artemis-cli-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110) [artemis-cli-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72) [artemis-cli-1.3.0.jar:1.3.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191]
at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122) [artemis-boot.jar:1.3.0]
at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45) [artemis-boot.jar:1.3.0]
Re: Artemis isn’t starting up
Posted by Roy Cohen <ro...@hotmail.com>.
If it helps I found this
https://access.redhat.com/documentation/en-us/red_hat_amq/7.5/html/release_notes_for_red_hat_amq_broker_7.5/resolved
[cid:2073AC6D-772E-494A-8F22-F78A0E33B04E]
And the below in our logs.
Which might explain what happened but not how to recover from this… :/
2022-05-11 14:54:39,868 INFO [org.apache.activemq.artemis.core.server] (main) AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2022-05-11 14:54:43,687 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,112 found and discarded
2022-05-11 14:54:43,690 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,137 found and discarded
2022-05-11 14:54:43,690 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,143 found and discarded
2022-05-11 14:54:43,691 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,158 found and discarded
2022-05-11 14:54:43,691 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,171 found and discarded
2022-05-11 14:54:43,691 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,172 found and discarded
2022-05-11 14:54:43,692 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,167 found and discarded
2022-05-11 14:54:43,692 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,168 found and discarded
2022-05-11 14:54:43,692 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,165 found and discarded
2022-05-11 14:54:43,692 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,164 found and discarded
2022-05-11 14:54:43,693 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,162 found and discarded
2022-05-11 14:54:43,693 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,163 found and discarded
2022-05-11 14:54:43,693 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,175 found and discarded
2022-05-11 14:54:43,693 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,173 found and discarded
2022-05-11 14:54:43,694 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,169 found and discarded
2022-05-11 14:54:43,694 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,189 found and discarded
2022-05-11 14:54:43,694 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,183 found and discarded
2022-05-11 14:54:43,694 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,181 found and discarded
2022-05-11 14:54:43,695 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,187 found and discarded
2022-05-11 14:54:43,695 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,188 found and discarded
2022-05-11 14:54:43,695 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,184 found and discarded
2022-05-11 14:54:43,695 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,185 found and discarded
2022-05-11 14:54:43,696 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,170 found and discarded
2022-05-11 14:54:43,696 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,176 found and discarded
2022-05-11 14:54:43,696 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,178 found and discarded
2022-05-11 14:54:43,696 WARN [org.apache.activemq.artemis.journal] (main) AMQ142015: Uncommitted transaction with id 2,478,942,179 found and discarded
2022-05-11 14:54:43,751 WARN [org.apache.activemq.artemis.core.server] (main) AMQ222043: Message for queue 128 which does not exist. This message will be ignored.
2022-05-11 14:54:43,751 WARN [org.apache.activemq.artemis.core.server] (main) AMQ222043: Message for queue 4,416 which does not exist. This message will be ignored.
2022-05-11 14:54:43,752 WARN [org.apache.activemq.artemis.core.server] (main) AMQ222043: Message for queue 2,368,423,659 which does not exist. This message will be ignored.
2022-05-11 14:54:43,752 WARN [org.apache.activemq.artemis.core.server] (main) AMQ222043: Message for queue 2,787 which does not exist. This message will be ignored.
2022-05-11 14:54:43,752 WARN [org.apache.activemq.artemis.core.server] (main) AMQ222043: Message for queue 80 which does not exist. This message will be ignored.
2022-05-11 14:54:43,752 WARN [org.apache.activemq.artemis.core.server] (main) AMQ222043: Message for queue 48 which does not exist. This message will be ignored.
On 11 May 2022, at 23:34, Roy Cohen <ro...@hotmail.com>> wrote:
Hi All
We have an urgent situation with our Artemis brokers pair in production which somehow has left 01 without any DLQs and 02 in a state where it’s not starting up.
The one which isn’t starting up is throwing this Exception, unfortunately I haven’t been able to find much online. Any help would be greatly appreciated
2022-05-11 20:35:38,673 ERROR [org.apache.activemq.artemis.core.server] (main) AMQ224000: Failure in initialisation: java.lang.IllegalStateException: Cannot find queue with id 80
at org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427) [artemis-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394) [artemis-jms-server-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65) [artemis-cli-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83) [artemis-cli-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110) [artemis-cli-1.3.0.jar:1.3.0]
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72) [artemis-cli-1.3.0.jar:1.3.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191]
at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122) [artemis-boot.jar:1.3.0]
at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45) [artemis-boot.jar:1.3.0]
Re: Artemis isn’t starting up
Posted by Roy Cohen <ro...@hotmail.com>.
On 13 May 2022, at 20:47, Justin Bertram <jb...@apache.org> wrote:
So we were thinking d that and append files from the live journal on top ?
I don't understand what you're asking here.
If my understanding is correct and the journal is always appending then I’m theory getting its co located backup from the other broker plus its current journal minus the file which causes it to crash should get us almost to where we are ? The reason why I’m asking this is because removing the offending file alone from the journal leaving me with zero messages, specially the DLQs which we need. However by taking the approach above judging by the number of messages in the queue plus DLQ it appears that we are almost close to where we were before the crash has happened (which by the way was caused by an unexpected data centre hardware fault).
I keep seeing journal and journal1 directories whereas broker.xml has
journal configured, any idea what is journal1 for ?
I'm not sure. If you're using colocation with replication then that might
be a replica backup caused by failing over and then stopping the backup
before it was able to fail-back. In that situation when the backup restarts
it will move its existing journal to a backup directory and then re-sync
with the live broker.
Yes we are using collocation with replication. So in theory if the above is correct then that directory is sort of a temp directory which should be left alone ?
Out of curiosity, did you rule out patching the broker? If so, why?
Because despite the fact it is a two lines change the efforts of patching, building, publishing, testing and getting it through the many environments is almost just as big as an upgrade. Even though technically speaking we can just patch prod where it failed and replace the binary that’s not how it should be done. I can only hope that with the current failure we’ve had a proper upgrade will be deemed as a higher priority…
Justin
On Fri, May 13, 2022 at 8:59 AM Roy Cohen <ro...@hotmail.com> wrote:
Thanks again Justin.
Re ARTEMIS-2460 / Upgrade - Sigh and LOL at the same time :@)
So, I was able to find the offending amq file, when I remove it from the
journal the broker does start up just fine however all queues are showing
zero count, whereas with a colocated backup restore we at least got a
“snapshot” of the day of the backup. So we were thinking d that and append
files from the live journal on top ?
Another question: I keep seeing journal and journal1 directories whereas
broker.xml has journal configured, any idea what is journal1 for ?
On 13 May 2022, at 14:48, Justin Bertram <jb...@apache.org> wrote:
Would it be possible to just add more files to the journal from the
colocated backup to recover missing messages ?
If you had a backup broker then it's certainly possible you could replace
the offending journal with the journal from the backup. However, I
wouldn't
copy/replace individual journal files. You'd want to do the whole thing.
Alternatively is there a way to know in which journal file that
offending
prepared transaction is, accept the fact that journal file is gone and
have
it removed from journal ?
You might be able to comb through the output of the 'data print' command
to
find which file it's in, but I'm not certain even that would be
effective.
In my opinion patching the broker is by far the simplest option (assuming
you know Java). It's really just a couple of lines.
While I understand we are far behind sadly that is not something we can
do quickly.
I've worked in large organizations where upgrading things was difficult
so
I can empathize. That said, it's important to stay up-to-date if you're
relying on community (i.e. free) support. If you had commercial support
that would, of course, be a different story. You were commenting on
ARTEMIS-2460 [1] almost 6 months ago that you were looking at upgrading.
Justin
[1] https://issues.apache.org/jira/browse/ARTEMIS-2460
On Thu, May 12, 2022 at 6:05 PM Roy Cohen <ro...@hotmail.com> wrote:
Alternatively is there a way to know in which journal file that
offending
prepared transaction is, accept the fact that journal file is gone and
have
it removed from journal ?
On 12 May 2022, at 23:08, Roy Cohen <ro...@hotmail.com> wrote:
Thank you Justin.
While I understand we are far behind sadly that is not something we can
do
quickly.
I did manage to restore the broker using its colocated backup binding
from
the other broker and it is back up.
Question: Would it be possible to just add more files to the journal
from
the colocated backup to recover missing messages ?
On 12 May 2022, at 21:02, Justin Bertram <jb...@apache.org> wrote:
As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The
fix
is
pretty straightforward as it just ignores the prepared transaction
causing
the problem.
At the moment I see two ways for you to proceed.
1) Download the latest version of ActiveMQ Artemis [2], create a new
instance, copy your journal to the instance's, start that instance, and
then move messages from that instance back to your 1.3.0 broker (e.g.
via
Camel, JMS bridge, etc.).
2) Download the source-code for 1.3.0 [3], apply the fix [4], build the
artemis-server module (cd artemis-server; mvn clean install
-DskipTests),
deploy the fixed artemis-server-1.3.0.jar to your broker
Aside from that, I strongly recommend you upgrade your broker to the
latest
version. Version 1.3.0 was released in June 2016, almost 6 years ago
now.
There's been 36 releases since then.
Justin
[1] https://issues.apache.org/jira/browse/ARTEMIS-1764
[2] https://activemq.apache.org/components/artemis/download/
[3]
https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz
[4]
https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5
On Wed, May 11, 2022 at 5:35 PM Roy Cohen <ro...@hotmail.com>
wrote:
Hi All
We have an urgent situation with our Artemis brokers pair in
production
which somehow has left 01 without any DLQs and 02 in a state where
it’s
not
starting up.
The one which isn’t starting up is throwing this Exception,
unfortunately
I haven’t been able to find much online. Any help would be greatly
appreciated
2022-05-11 20:35:38,673 ERROR
[org.apache.activemq.artemis.core.server]
(main) AMQ224000: Failure in initialisation:
java.lang.IllegalStateException: Cannot find queue with id 80
at
org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281)
[artemis-server-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649)
[artemis-server-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167)
[artemis-server-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054)
[artemis-server-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927)
[artemis-server-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100)
[artemis-server-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108)
[artemis-server-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427)
[artemis-server-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394)
[artemis-jms-server-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65)
[artemis-cli-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83)
[artemis-cli-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110)
[artemis-cli-1.3.0.jar:1.3.0]
at
org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72)
[artemis-cli-1.3.0.jar:1.3.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_191]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_191]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498)
[rt.jar:1.8.0_191]
at
org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122)
[artemis-boot.jar:1.3.0]
at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45)
[artemis-boot.jar:1.3.0]
Re: Artemis isn’t starting up
Posted by Justin Bertram <jb...@apache.org>.
> So we were thinking d that and append files from the live journal on top ?
I don't understand what you're asking here.
> I keep seeing journal and journal1 directories whereas broker.xml has
journal configured, any idea what is journal1 for ?
I'm not sure. If you're using colocation with replication then that might
be a replica backup caused by failing over and then stopping the backup
before it was able to fail-back. In that situation when the backup restarts
it will move its existing journal to a backup directory and then re-sync
with the live broker.
Out of curiosity, did you rule out patching the broker? If so, why?
Justin
On Fri, May 13, 2022 at 8:59 AM Roy Cohen <ro...@hotmail.com> wrote:
> Thanks again Justin.
>
> Re ARTEMIS-2460 / Upgrade - Sigh and LOL at the same time :@)
>
> So, I was able to find the offending amq file, when I remove it from the
> journal the broker does start up just fine however all queues are showing
> zero count, whereas with a colocated backup restore we at least got a
> “snapshot” of the day of the backup. So we were thinking d that and append
> files from the live journal on top ?
>
> Another question: I keep seeing journal and journal1 directories whereas
> broker.xml has journal configured, any idea what is journal1 for ?
>
>
>
> > On 13 May 2022, at 14:48, Justin Bertram <jb...@apache.org> wrote:
> >
> >> Would it be possible to just add more files to the journal from the
> > colocated backup to recover missing messages ?
> >
> > If you had a backup broker then it's certainly possible you could replace
> > the offending journal with the journal from the backup. However, I
> wouldn't
> > copy/replace individual journal files. You'd want to do the whole thing.
> >
> >> Alternatively is there a way to know in which journal file that
> offending
> > prepared transaction is, accept the fact that journal file is gone and
> have
> > it removed from journal ?
> >
> > You might be able to comb through the output of the 'data print' command
> to
> > find which file it's in, but I'm not certain even that would be
> effective.
> >
> > In my opinion patching the broker is by far the simplest option (assuming
> > you know Java). It's really just a couple of lines.
> >
> >> While I understand we are far behind sadly that is not something we can
> > do quickly.
> >
> > I've worked in large organizations where upgrading things was difficult
> so
> > I can empathize. That said, it's important to stay up-to-date if you're
> > relying on community (i.e. free) support. If you had commercial support
> > that would, of course, be a different story. You were commenting on
> > ARTEMIS-2460 [1] almost 6 months ago that you were looking at upgrading.
> >
> >
> > Justin
> >
> > [1] https://issues.apache.org/jira/browse/ARTEMIS-2460
> >
> > On Thu, May 12, 2022 at 6:05 PM Roy Cohen <ro...@hotmail.com> wrote:
> >
> >> Alternatively is there a way to know in which journal file that
> offending
> >> prepared transaction is, accept the fact that journal file is gone and
> have
> >> it removed from journal ?
> >>
> >> On 12 May 2022, at 23:08, Roy Cohen <ro...@hotmail.com> wrote:
> >>
> >> Thank you Justin.
> >>
> >> While I understand we are far behind sadly that is not something we can
> do
> >> quickly.
> >>
> >> I did manage to restore the broker using its colocated backup binding
> from
> >> the other broker and it is back up.
> >>
> >> Question: Would it be possible to just add more files to the journal
> from
> >> the colocated backup to recover missing messages ?
> >>
> >>
> >>
> >>> On 12 May 2022, at 21:02, Justin Bertram <jb...@apache.org> wrote:
> >>>
> >>> As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The
> fix
> >> is
> >>> pretty straightforward as it just ignores the prepared transaction
> >> causing
> >>> the problem.
> >>>
> >>> At the moment I see two ways for you to proceed.
> >>>
> >>> 1) Download the latest version of ActiveMQ Artemis [2], create a new
> >>> instance, copy your journal to the instance's, start that instance, and
> >>> then move messages from that instance back to your 1.3.0 broker (e.g.
> via
> >>> Camel, JMS bridge, etc.).
> >>> 2) Download the source-code for 1.3.0 [3], apply the fix [4], build the
> >>> artemis-server module (cd artemis-server; mvn clean install
> -DskipTests),
> >>> deploy the fixed artemis-server-1.3.0.jar to your broker
> >>>
> >>> Aside from that, I strongly recommend you upgrade your broker to the
> >> latest
> >>> version. Version 1.3.0 was released in June 2016, almost 6 years ago
> now.
> >>> There's been 36 releases since then.
> >>>
> >>>
> >>> Justin
> >>>
> >>> [1] https://issues.apache.org/jira/browse/ARTEMIS-1764
> >>> [2] https://activemq.apache.org/components/artemis/download/
> >>> [3]
> >>>
> >>
> https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz
> >>> [4]
> >>>
> >>
> https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5
> >>>
> >>>
> >>> On Wed, May 11, 2022 at 5:35 PM Roy Cohen <ro...@hotmail.com>
> wrote:
> >>>
> >>>> Hi All
> >>>>
> >>>> We have an urgent situation with our Artemis brokers pair in
> production
> >>>> which somehow has left 01 without any DLQs and 02 in a state where
> it’s
> >> not
> >>>> starting up.
> >>>>
> >>>> The one which isn’t starting up is throwing this Exception,
> >> unfortunately
> >>>> I haven’t been able to find much online. Any help would be greatly
> >>>> appreciated
> >>>>
> >>>> 2022-05-11 20:35:38,673 ERROR
> [org.apache.activemq.artemis.core.server]
> >>>> (main) AMQ224000: Failure in initialisation:
> >>>> java.lang.IllegalStateException: Cannot find queue with id 80
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281)
> >>>> [artemis-server-1.3.0.jar:1.3.0]
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649)
> >>>> [artemis-server-1.3.0.jar:1.3.0]
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167)
> >>>> [artemis-server-1.3.0.jar:1.3.0]
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054)
> >>>> [artemis-server-1.3.0.jar:1.3.0]
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927)
> >>>> [artemis-server-1.3.0.jar:1.3.0]
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100)
> >>>> [artemis-server-1.3.0.jar:1.3.0]
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108)
> >>>> [artemis-server-1.3.0.jar:1.3.0]
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427)
> >>>> [artemis-server-1.3.0.jar:1.3.0]
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394)
> >>>> [artemis-jms-server-1.3.0.jar:1.3.0]
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65)
> >>>> [artemis-cli-1.3.0.jar:1.3.0]
> >>>> at
> >>>> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83)
> >>>> [artemis-cli-1.3.0.jar:1.3.0]
> >>>> at
> >>>>
> >>
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110)
> >>>> [artemis-cli-1.3.0.jar:1.3.0]
> >>>> at
> >>>> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72)
> >>>> [artemis-cli-1.3.0.jar:1.3.0]
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> [rt.jar:1.8.0_191]
> >>>> at
> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>>> [rt.jar:1.8.0_191]
> >>>> at
> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> [rt.jar:1.8.0_191]
> >>>> at java.lang.reflect.Method.invoke(Method.java:498)
> >>>> [rt.jar:1.8.0_191]
> >>>> at
> >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122)
> >>>> [artemis-boot.jar:1.3.0]
> >>>> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45)
> >>>> [artemis-boot.jar:1.3.0]
> >>>>
> >>>>
> >>
> >>
>
>
Re: Artemis isn’t starting up
Posted by Roy Cohen <ro...@hotmail.com>.
Thanks again Justin.
Re ARTEMIS-2460 / Upgrade - Sigh and LOL at the same time :@)
So, I was able to find the offending amq file, when I remove it from the journal the broker does start up just fine however all queues are showing zero count, whereas with a colocated backup restore we at least got a “snapshot” of the day of the backup. So we were thinking d that and append files from the live journal on top ?
Another question: I keep seeing journal and journal1 directories whereas broker.xml has journal configured, any idea what is journal1 for ?
> On 13 May 2022, at 14:48, Justin Bertram <jb...@apache.org> wrote:
>
>> Would it be possible to just add more files to the journal from the
> colocated backup to recover missing messages ?
>
> If you had a backup broker then it's certainly possible you could replace
> the offending journal with the journal from the backup. However, I wouldn't
> copy/replace individual journal files. You'd want to do the whole thing.
>
>> Alternatively is there a way to know in which journal file that offending
> prepared transaction is, accept the fact that journal file is gone and have
> it removed from journal ?
>
> You might be able to comb through the output of the 'data print' command to
> find which file it's in, but I'm not certain even that would be effective.
>
> In my opinion patching the broker is by far the simplest option (assuming
> you know Java). It's really just a couple of lines.
>
>> While I understand we are far behind sadly that is not something we can
> do quickly.
>
> I've worked in large organizations where upgrading things was difficult so
> I can empathize. That said, it's important to stay up-to-date if you're
> relying on community (i.e. free) support. If you had commercial support
> that would, of course, be a different story. You were commenting on
> ARTEMIS-2460 [1] almost 6 months ago that you were looking at upgrading.
>
>
> Justin
>
> [1] https://issues.apache.org/jira/browse/ARTEMIS-2460
>
> On Thu, May 12, 2022 at 6:05 PM Roy Cohen <ro...@hotmail.com> wrote:
>
>> Alternatively is there a way to know in which journal file that offending
>> prepared transaction is, accept the fact that journal file is gone and have
>> it removed from journal ?
>>
>> On 12 May 2022, at 23:08, Roy Cohen <ro...@hotmail.com> wrote:
>>
>> Thank you Justin.
>>
>> While I understand we are far behind sadly that is not something we can do
>> quickly.
>>
>> I did manage to restore the broker using its colocated backup binding from
>> the other broker and it is back up.
>>
>> Question: Would it be possible to just add more files to the journal from
>> the colocated backup to recover missing messages ?
>>
>>
>>
>>> On 12 May 2022, at 21:02, Justin Bertram <jb...@apache.org> wrote:
>>>
>>> As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The fix
>> is
>>> pretty straightforward as it just ignores the prepared transaction
>> causing
>>> the problem.
>>>
>>> At the moment I see two ways for you to proceed.
>>>
>>> 1) Download the latest version of ActiveMQ Artemis [2], create a new
>>> instance, copy your journal to the instance's, start that instance, and
>>> then move messages from that instance back to your 1.3.0 broker (e.g. via
>>> Camel, JMS bridge, etc.).
>>> 2) Download the source-code for 1.3.0 [3], apply the fix [4], build the
>>> artemis-server module (cd artemis-server; mvn clean install -DskipTests),
>>> deploy the fixed artemis-server-1.3.0.jar to your broker
>>>
>>> Aside from that, I strongly recommend you upgrade your broker to the
>> latest
>>> version. Version 1.3.0 was released in June 2016, almost 6 years ago now.
>>> There's been 36 releases since then.
>>>
>>>
>>> Justin
>>>
>>> [1] https://issues.apache.org/jira/browse/ARTEMIS-1764
>>> [2] https://activemq.apache.org/components/artemis/download/
>>> [3]
>>>
>> https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz
>>> [4]
>>>
>> https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5
>>>
>>>
>>> On Wed, May 11, 2022 at 5:35 PM Roy Cohen <ro...@hotmail.com> wrote:
>>>
>>>> Hi All
>>>>
>>>> We have an urgent situation with our Artemis brokers pair in production
>>>> which somehow has left 01 without any DLQs and 02 in a state where it’s
>> not
>>>> starting up.
>>>>
>>>> The one which isn’t starting up is throwing this Exception,
>> unfortunately
>>>> I haven’t been able to find much online. Any help would be greatly
>>>> appreciated
>>>>
>>>> 2022-05-11 20:35:38,673 ERROR [org.apache.activemq.artemis.core.server]
>>>> (main) AMQ224000: Failure in initialisation:
>>>> java.lang.IllegalStateException: Cannot find queue with id 80
>>>> at
>>>>
>> org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281)
>>>> [artemis-server-1.3.0.jar:1.3.0]
>>>> at
>>>>
>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649)
>>>> [artemis-server-1.3.0.jar:1.3.0]
>>>> at
>>>>
>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167)
>>>> [artemis-server-1.3.0.jar:1.3.0]
>>>> at
>>>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054)
>>>> [artemis-server-1.3.0.jar:1.3.0]
>>>> at
>>>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927)
>>>> [artemis-server-1.3.0.jar:1.3.0]
>>>> at
>>>>
>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100)
>>>> [artemis-server-1.3.0.jar:1.3.0]
>>>> at
>>>>
>> org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108)
>>>> [artemis-server-1.3.0.jar:1.3.0]
>>>> at
>>>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427)
>>>> [artemis-server-1.3.0.jar:1.3.0]
>>>> at
>>>>
>> org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394)
>>>> [artemis-jms-server-1.3.0.jar:1.3.0]
>>>> at
>>>>
>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65)
>>>> [artemis-cli-1.3.0.jar:1.3.0]
>>>> at
>>>> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83)
>>>> [artemis-cli-1.3.0.jar:1.3.0]
>>>> at
>>>>
>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110)
>>>> [artemis-cli-1.3.0.jar:1.3.0]
>>>> at
>>>> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72)
>>>> [artemis-cli-1.3.0.jar:1.3.0]
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> [rt.jar:1.8.0_191]
>>>> at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> [rt.jar:1.8.0_191]
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> [rt.jar:1.8.0_191]
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> [rt.jar:1.8.0_191]
>>>> at
>>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122)
>>>> [artemis-boot.jar:1.3.0]
>>>> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45)
>>>> [artemis-boot.jar:1.3.0]
>>>>
>>>>
>>
>>
Re: Artemis isn’t starting up
Posted by Justin Bertram <jb...@apache.org>.
> Would it be possible to just add more files to the journal from the
colocated backup to recover missing messages ?
If you had a backup broker then it's certainly possible you could replace
the offending journal with the journal from the backup. However, I wouldn't
copy/replace individual journal files. You'd want to do the whole thing.
> Alternatively is there a way to know in which journal file that offending
prepared transaction is, accept the fact that journal file is gone and have
it removed from journal ?
You might be able to comb through the output of the 'data print' command to
find which file it's in, but I'm not certain even that would be effective.
In my opinion patching the broker is by far the simplest option (assuming
you know Java). It's really just a couple of lines.
> While I understand we are far behind sadly that is not something we can
do quickly.
I've worked in large organizations where upgrading things was difficult so
I can empathize. That said, it's important to stay up-to-date if you're
relying on community (i.e. free) support. If you had commercial support
that would, of course, be a different story. You were commenting on
ARTEMIS-2460 [1] almost 6 months ago that you were looking at upgrading.
Justin
[1] https://issues.apache.org/jira/browse/ARTEMIS-2460
On Thu, May 12, 2022 at 6:05 PM Roy Cohen <ro...@hotmail.com> wrote:
> Alternatively is there a way to know in which journal file that offending
> prepared transaction is, accept the fact that journal file is gone and have
> it removed from journal ?
>
> On 12 May 2022, at 23:08, Roy Cohen <ro...@hotmail.com> wrote:
>
> Thank you Justin.
>
> While I understand we are far behind sadly that is not something we can do
> quickly.
>
> I did manage to restore the broker using its colocated backup binding from
> the other broker and it is back up.
>
> Question: Would it be possible to just add more files to the journal from
> the colocated backup to recover missing messages ?
>
>
>
> > On 12 May 2022, at 21:02, Justin Bertram <jb...@apache.org> wrote:
> >
> > As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The fix
> is
> > pretty straightforward as it just ignores the prepared transaction
> causing
> > the problem.
> >
> > At the moment I see two ways for you to proceed.
> >
> > 1) Download the latest version of ActiveMQ Artemis [2], create a new
> > instance, copy your journal to the instance's, start that instance, and
> > then move messages from that instance back to your 1.3.0 broker (e.g. via
> > Camel, JMS bridge, etc.).
> > 2) Download the source-code for 1.3.0 [3], apply the fix [4], build the
> > artemis-server module (cd artemis-server; mvn clean install -DskipTests),
> > deploy the fixed artemis-server-1.3.0.jar to your broker
> >
> > Aside from that, I strongly recommend you upgrade your broker to the
> latest
> > version. Version 1.3.0 was released in June 2016, almost 6 years ago now.
> > There's been 36 releases since then.
> >
> >
> > Justin
> >
> > [1] https://issues.apache.org/jira/browse/ARTEMIS-1764
> > [2] https://activemq.apache.org/components/artemis/download/
> > [3]
> >
> https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz
> > [4]
> >
> https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5
> >
> >
> > On Wed, May 11, 2022 at 5:35 PM Roy Cohen <ro...@hotmail.com> wrote:
> >
> >> Hi All
> >>
> >> We have an urgent situation with our Artemis brokers pair in production
> >> which somehow has left 01 without any DLQs and 02 in a state where it’s
> not
> >> starting up.
> >>
> >> The one which isn’t starting up is throwing this Exception,
> unfortunately
> >> I haven’t been able to find much online. Any help would be greatly
> >> appreciated
> >>
> >> 2022-05-11 20:35:38,673 ERROR [org.apache.activemq.artemis.core.server]
> >> (main) AMQ224000: Failure in initialisation:
> >> java.lang.IllegalStateException: Cannot find queue with id 80
> >> at
> >>
> org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281)
> >> [artemis-server-1.3.0.jar:1.3.0]
> >> at
> >>
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649)
> >> [artemis-server-1.3.0.jar:1.3.0]
> >> at
> >>
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167)
> >> [artemis-server-1.3.0.jar:1.3.0]
> >> at
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054)
> >> [artemis-server-1.3.0.jar:1.3.0]
> >> at
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927)
> >> [artemis-server-1.3.0.jar:1.3.0]
> >> at
> >>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100)
> >> [artemis-server-1.3.0.jar:1.3.0]
> >> at
> >>
> org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108)
> >> [artemis-server-1.3.0.jar:1.3.0]
> >> at
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427)
> >> [artemis-server-1.3.0.jar:1.3.0]
> >> at
> >>
> org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394)
> >> [artemis-jms-server-1.3.0.jar:1.3.0]
> >> at
> >>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65)
> >> [artemis-cli-1.3.0.jar:1.3.0]
> >> at
> >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83)
> >> [artemis-cli-1.3.0.jar:1.3.0]
> >> at
> >>
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110)
> >> [artemis-cli-1.3.0.jar:1.3.0]
> >> at
> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72)
> >> [artemis-cli-1.3.0.jar:1.3.0]
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> [rt.jar:1.8.0_191]
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> [rt.jar:1.8.0_191]
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> [rt.jar:1.8.0_191]
> >> at java.lang.reflect.Method.invoke(Method.java:498)
> >> [rt.jar:1.8.0_191]
> >> at
> >> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122)
> >> [artemis-boot.jar:1.3.0]
> >> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45)
> >> [artemis-boot.jar:1.3.0]
> >>
> >>
>
>
Re: Artemis isn’t starting up
Posted by Roy Cohen <ro...@hotmail.com>.
Alternatively is there a way to know in which journal file that offending prepared transaction is, accept the fact that journal file is gone and have it removed from journal ?
On 12 May 2022, at 23:08, Roy Cohen <ro...@hotmail.com> wrote:
Thank you Justin.
While I understand we are far behind sadly that is not something we can do quickly.
I did manage to restore the broker using its colocated backup binding from the other broker and it is back up.
Question: Would it be possible to just add more files to the journal from the colocated backup to recover missing messages ?
> On 12 May 2022, at 21:02, Justin Bertram <jb...@apache.org> wrote:
>
> As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The fix is
> pretty straightforward as it just ignores the prepared transaction causing
> the problem.
>
> At the moment I see two ways for you to proceed.
>
> 1) Download the latest version of ActiveMQ Artemis [2], create a new
> instance, copy your journal to the instance's, start that instance, and
> then move messages from that instance back to your 1.3.0 broker (e.g. via
> Camel, JMS bridge, etc.).
> 2) Download the source-code for 1.3.0 [3], apply the fix [4], build the
> artemis-server module (cd artemis-server; mvn clean install -DskipTests),
> deploy the fixed artemis-server-1.3.0.jar to your broker
>
> Aside from that, I strongly recommend you upgrade your broker to the latest
> version. Version 1.3.0 was released in June 2016, almost 6 years ago now.
> There's been 36 releases since then.
>
>
> Justin
>
> [1] https://issues.apache.org/jira/browse/ARTEMIS-1764
> [2] https://activemq.apache.org/components/artemis/download/
> [3]
> https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz
> [4]
> https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5
>
>
> On Wed, May 11, 2022 at 5:35 PM Roy Cohen <ro...@hotmail.com> wrote:
>
>> Hi All
>>
>> We have an urgent situation with our Artemis brokers pair in production
>> which somehow has left 01 without any DLQs and 02 in a state where it’s not
>> starting up.
>>
>> The one which isn’t starting up is throwing this Exception, unfortunately
>> I haven’t been able to find much online. Any help would be greatly
>> appreciated
>>
>> 2022-05-11 20:35:38,673 ERROR [org.apache.activemq.artemis.core.server]
>> (main) AMQ224000: Failure in initialisation:
>> java.lang.IllegalStateException: Cannot find queue with id 80
>> at
>> org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394)
>> [artemis-jms-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65)
>> [artemis-cli-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83)
>> [artemis-cli-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110)
>> [artemis-cli-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72)
>> [artemis-cli-1.3.0.jar:1.3.0]
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> [rt.jar:1.8.0_191]
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> [rt.jar:1.8.0_191]
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> [rt.jar:1.8.0_191]
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> [rt.jar:1.8.0_191]
>> at
>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122)
>> [artemis-boot.jar:1.3.0]
>> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45)
>> [artemis-boot.jar:1.3.0]
>>
>>
Re: Artemis isn’t starting up
Posted by Roy Cohen <ro...@hotmail.com>.
Thank you Justin.
While I understand we are far behind sadly that is not something we can do quickly.
I did manage to restore the broker using its colocated backup binding from the other broker and it is back up.
Question: Would it be possible to just add more files to the journal from the colocated backup to recover missing messages ?
> On 12 May 2022, at 21:02, Justin Bertram <jb...@apache.org> wrote:
>
> As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The fix is
> pretty straightforward as it just ignores the prepared transaction causing
> the problem.
>
> At the moment I see two ways for you to proceed.
>
> 1) Download the latest version of ActiveMQ Artemis [2], create a new
> instance, copy your journal to the instance's, start that instance, and
> then move messages from that instance back to your 1.3.0 broker (e.g. via
> Camel, JMS bridge, etc.).
> 2) Download the source-code for 1.3.0 [3], apply the fix [4], build the
> artemis-server module (cd artemis-server; mvn clean install -DskipTests),
> deploy the fixed artemis-server-1.3.0.jar to your broker
>
> Aside from that, I strongly recommend you upgrade your broker to the latest
> version. Version 1.3.0 was released in June 2016, almost 6 years ago now.
> There's been 36 releases since then.
>
>
> Justin
>
> [1] https://issues.apache.org/jira/browse/ARTEMIS-1764
> [2] https://activemq.apache.org/components/artemis/download/
> [3]
> https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz
> [4]
> https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5
>
>
> On Wed, May 11, 2022 at 5:35 PM Roy Cohen <ro...@hotmail.com> wrote:
>
>> Hi All
>>
>> We have an urgent situation with our Artemis brokers pair in production
>> which somehow has left 01 without any DLQs and 02 in a state where it’s not
>> starting up.
>>
>> The one which isn’t starting up is throwing this Exception, unfortunately
>> I haven’t been able to find much online. Any help would be greatly
>> appreciated
>>
>> 2022-05-11 20:35:38,673 ERROR [org.apache.activemq.artemis.core.server]
>> (main) AMQ224000: Failure in initialisation:
>> java.lang.IllegalStateException: Cannot find queue with id 80
>> at
>> org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427)
>> [artemis-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394)
>> [artemis-jms-server-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65)
>> [artemis-cli-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83)
>> [artemis-cli-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110)
>> [artemis-cli-1.3.0.jar:1.3.0]
>> at
>> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72)
>> [artemis-cli-1.3.0.jar:1.3.0]
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> [rt.jar:1.8.0_191]
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> [rt.jar:1.8.0_191]
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> [rt.jar:1.8.0_191]
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> [rt.jar:1.8.0_191]
>> at
>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122)
>> [artemis-boot.jar:1.3.0]
>> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45)
>> [artemis-boot.jar:1.3.0]
>>
>>
Re: Artemis isn’t starting up
Posted by Justin Bertram <jb...@apache.org>.
As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The fix is
pretty straightforward as it just ignores the prepared transaction causing
the problem.
At the moment I see two ways for you to proceed.
1) Download the latest version of ActiveMQ Artemis [2], create a new
instance, copy your journal to the instance's, start that instance, and
then move messages from that instance back to your 1.3.0 broker (e.g. via
Camel, JMS bridge, etc.).
2) Download the source-code for 1.3.0 [3], apply the fix [4], build the
artemis-server module (cd artemis-server; mvn clean install -DskipTests),
deploy the fixed artemis-server-1.3.0.jar to your broker
Aside from that, I strongly recommend you upgrade your broker to the latest
version. Version 1.3.0 was released in June 2016, almost 6 years ago now.
There's been 36 releases since then.
Justin
[1] https://issues.apache.org/jira/browse/ARTEMIS-1764
[2] https://activemq.apache.org/components/artemis/download/
[3]
https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz
[4]
https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5
On Wed, May 11, 2022 at 5:35 PM Roy Cohen <ro...@hotmail.com> wrote:
> Hi All
>
> We have an urgent situation with our Artemis brokers pair in production
> which somehow has left 01 without any DLQs and 02 in a state where it’s not
> starting up.
>
> The one which isn’t starting up is throwing this Exception, unfortunately
> I haven’t been able to find much online. Any help would be greatly
> appreciated
>
> 2022-05-11 20:35:38,673 ERROR [org.apache.activemq.artemis.core.server]
> (main) AMQ224000: Failure in initialisation:
> java.lang.IllegalStateException: Cannot find queue with id 80
> at
> org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281)
> [artemis-server-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649)
> [artemis-server-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167)
> [artemis-server-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054)
> [artemis-server-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927)
> [artemis-server-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100)
> [artemis-server-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108)
> [artemis-server-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427)
> [artemis-server-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394)
> [artemis-jms-server-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65)
> [artemis-cli-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83)
> [artemis-cli-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110)
> [artemis-cli-1.3.0.jar:1.3.0]
> at
> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72)
> [artemis-cli-1.3.0.jar:1.3.0]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [rt.jar:1.8.0_191]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [rt.jar:1.8.0_191]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_191]
> at java.lang.reflect.Method.invoke(Method.java:498)
> [rt.jar:1.8.0_191]
> at
> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122)
> [artemis-boot.jar:1.3.0]
> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45)
> [artemis-boot.jar:1.3.0]
>
>