You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by ch...@baus.net on 2017/05/19 18:32:40 UTC
Migrating storage from 5.10.0 to 5.14.5
Good Day,
I've been working on migrating servers and upgrading from 5.10.0
to 5.14.5.
We have a number of scheduled tasks and I want to preserve those along
with other items in the queue. I tried moving the files in kahadb to the
new server (after shutting down both servers), but have run into
multiple errors which prevent the new server from starting.
I am wondering what the appropriate way to migrate storage is. Should I
attempt read all of the messages from the old queue and write them into
the new queue? Has anyone else done this before? It seems like this
would be a useful utility.
Thanks,
Chris
Re: Migrating storage from 5.10.0 to 5.14.5
Posted by mikmela <mi...@yahoo.com>.
I don't think there's any current fallback option for SQL right now.
Contributions are welcome though so if you wanted to take a shot at adding
a property for that you could create a PR.
</quot>
We might give it a shot - should we follow
https://activemq.apache.org/contributing.html and create a patch or just
pull from
https://github.com/apache/activemq?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Migrating storage from 5.10.0 to 5.14.5
Posted by Christopher Shannon <ch...@gmail.com>.
I don't think there's any current fallback option for SQL right now.
Contributions are welcome though so if you wanted to take a shot at adding
a property for that you could create a PR.
On Wed, Apr 17, 2019 at 4:28 PM mikmela <mi...@yahoo.com> wrote:
> We seems to experience the same issue upgrading from 5.6.0 to 515.8, but in
> our case we use SQL database as a message store. As it was mentioned and
> confirmed by us KahaDB has a fallback capability when it encounter older
> (Version 6) of OpenWire.
> Unfortunately, it looks like for SQL databases there is no fallback option
> implemented:
> 2019-04-17 15:50:26,361 ERROR [Thread-8] (BrokerService.java:639) - Failed
> to start Apache ActiveMQ (RTCV180R1GPWR78QA12T3DEV-NY-CONNEX61616, null)
> java.io.UTFDataFormatException: bad string
> at
>
> org.apache.activemq.util.DataByteArrayInputStream.readUTF(DataByteArrayInputStream.java:265)
> at
>
> org.apache.activemq.openwire.v11.BaseDataStreamMarshaller.looseUnmarshalString(BaseDataStreamMarshaller.java:571)
> at
>
> org.apache.activemq.openwire.v11.MessageIdMarshaller.looseUnmarshal(MessageIdMarshaller.java:122)
> at
>
> org.apache.activemq.openwire.OpenWireFormat.looseUnmarshalNestedObject(OpenWireFormat.java:474)
> at
>
> org.apache.activemq.openwire.v11.BaseDataStreamMarshaller.looseUnmarsalNestedObject(BaseDataStreamMarshaller.java:466)
> at
>
> org.apache.activemq.openwire.v11.MessageMarshaller.looseUnmarshal(MessageMarshaller.java:220)
> at
>
> org.apache.activemq.openwire.v11.ActiveMQMessageMarshaller.looseUnmarshal(ActiveMQMessageMarshaller.java:101)
> at
>
> org.apache.activemq.openwire.v11.ActiveMQTextMessageMarshaller.looseUnmarshal(ActiveMQTextMessageMarshaller.java:101)
> at
>
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:367)
> at
>
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:201)
> at
>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getLastMessageBrokerSequenceId(JDBCPersistenceAdapter.java:266)
> at
>
> org.apache.activemq.broker.region.DestinationFactoryImpl.getLastMessageBrokerSequenceId(DestinationFactoryImpl.java:147)
> at
>
> org.apache.activemq.broker.region.RegionBroker.<init>(RegionBroker.java:130)
> at
>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.<init>(ManagedRegionBroker.java:108)
> at
>
> org.apache.activemq.broker.BrokerService.createRegionBroker(BrokerService.java:2399)
> at
>
> org.apache.activemq.broker.BrokerService.createRegionBroker(BrokerService.java:2391)
> at
>
> org.apache.activemq.broker.BrokerService.createBroker(BrokerService.java:2348)
> at
> org.apache.activemq.broker.BrokerService.getBroker(BrokerService.java:1045)
> at
>
> org.apache.activemq.broker.BrokerService.getAdminConnectionContext(BrokerService.java:2619)
> at
>
> org.apache.activemq.broker.BrokerService.startVirtualConsumerDestinations(BrokerService.java:2780)
> at
>
> org.apache.activemq.broker.BrokerService.startDestinations(BrokerService.java:2610)
> at
>
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:739)
> at
>
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:733)
> at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:636)
> at
>
> COM.olf.adapter.jbroker.JBrokerService.startBroker(JBrokerService.java:1507)
> at
> COM.olf.adapter.jbroker.JBrokerComponent.run(JBrokerComponent.java:220)
> at java.lang.Thread.run(Thread.java:748)
>
> I'm wondering if there is a way to enable backward compatibility via some
> kind of property, or there are some other ways to do it?
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Migrating storage from 5.10.0 to 5.14.5
Posted by mikmela <mi...@yahoo.com>.
We seems to experience the same issue upgrading from 5.6.0 to 515.8, but in
our case we use SQL database as a message store. As it was mentioned and
confirmed by us KahaDB has a fallback capability when it encounter older
(Version 6) of OpenWire.
Unfortunately, it looks like for SQL databases there is no fallback option
implemented:
2019-04-17 15:50:26,361 ERROR [Thread-8] (BrokerService.java:639) - Failed
to start Apache ActiveMQ (RTCV180R1GPWR78QA12T3DEV-NY-CONNEX61616, null)
java.io.UTFDataFormatException: bad string
at
org.apache.activemq.util.DataByteArrayInputStream.readUTF(DataByteArrayInputStream.java:265)
at
org.apache.activemq.openwire.v11.BaseDataStreamMarshaller.looseUnmarshalString(BaseDataStreamMarshaller.java:571)
at
org.apache.activemq.openwire.v11.MessageIdMarshaller.looseUnmarshal(MessageIdMarshaller.java:122)
at
org.apache.activemq.openwire.OpenWireFormat.looseUnmarshalNestedObject(OpenWireFormat.java:474)
at
org.apache.activemq.openwire.v11.BaseDataStreamMarshaller.looseUnmarsalNestedObject(BaseDataStreamMarshaller.java:466)
at
org.apache.activemq.openwire.v11.MessageMarshaller.looseUnmarshal(MessageMarshaller.java:220)
at
org.apache.activemq.openwire.v11.ActiveMQMessageMarshaller.looseUnmarshal(ActiveMQMessageMarshaller.java:101)
at
org.apache.activemq.openwire.v11.ActiveMQTextMessageMarshaller.looseUnmarshal(ActiveMQTextMessageMarshaller.java:101)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:367)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:201)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getLastMessageBrokerSequenceId(JDBCPersistenceAdapter.java:266)
at
org.apache.activemq.broker.region.DestinationFactoryImpl.getLastMessageBrokerSequenceId(DestinationFactoryImpl.java:147)
at
org.apache.activemq.broker.region.RegionBroker.<init>(RegionBroker.java:130)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.<init>(ManagedRegionBroker.java:108)
at
org.apache.activemq.broker.BrokerService.createRegionBroker(BrokerService.java:2399)
at
org.apache.activemq.broker.BrokerService.createRegionBroker(BrokerService.java:2391)
at
org.apache.activemq.broker.BrokerService.createBroker(BrokerService.java:2348)
at
org.apache.activemq.broker.BrokerService.getBroker(BrokerService.java:1045)
at
org.apache.activemq.broker.BrokerService.getAdminConnectionContext(BrokerService.java:2619)
at
org.apache.activemq.broker.BrokerService.startVirtualConsumerDestinations(BrokerService.java:2780)
at
org.apache.activemq.broker.BrokerService.startDestinations(BrokerService.java:2610)
at
org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:739)
at
org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:733)
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:636)
at
COM.olf.adapter.jbroker.JBrokerService.startBroker(JBrokerService.java:1507)
at COM.olf.adapter.jbroker.JBrokerComponent.run(JBrokerComponent.java:220)
at java.lang.Thread.run(Thread.java:748)
I'm wondering if there is a way to enable backward compatibility via some
kind of property, or there are some other ways to do it?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Migrating storage from 5.10.0 to 5.14.5
Posted by ch...@baus.net.
The issue we have is that we have a number of scheduled tasks in the old
queue. What I ended up doing was writing a program to read out of the
old queue and then manually enqueue the tasks in the new queue. It was a
little awkward, but it got the job done.
On Mon, Jun 5, 2017, at 10:04 PM, Tim Bain wrote:
> What Chris described is how to load the existing messages into
> your new> broker. That has always seemed clunky to me because there is no way to> store messages in multiple OpenWire formats (which means your new
> broker's
> store is stuck forever on the old broker's OpenWire version), so the
> option
> that always sounded better to me is to stand up the new broker
> empty and> then attach the old broker in a network of brokers
> configuration. Clients> would connect to the new broker, and messages would flow from the old> broker to the new one and then to the consumers.
>
> On May 24, 2017 2:45 PM, "Christopher Shannon" <
> christopher.l.shannon@gmail.com> wrote:
>
>> What kind of errors are you getting? If it's related to data
>> marshalling>> then it could be because there was an Openwire upgrade
>> (Openwire is the>> format the messages are stored in) done in the KahaDB store
>> between those>> versions which could be causing your issue. I think there was
>> also some>> scheduler work done so not sure off the top of my head if that is the>> problem without seeing the stack trace.
>>
>> If you are simply copying over the journal files and trying to
>> rebuild the>> KahaDB index then it will fail as it will pick Openwire version 11
>> instead>> of 6. You set the version manually using the storeOpenWireVersion
>> property. See http://activemq.apache.org/kahadb.html. If the
>> index file>> still exists (db.data) then KahaDB should be able to detect that
>> it is an>> old store and fall back automatically to the write openwire version.>>
>>
>>
>> On Fri, May 19, 2017 at 2:32 PM, <ch...@baus.net> wrote:
>>
>>> Good Day,
>>>
>>> I've been working on migrating servers and upgrading from 5.10.0
>>> to 5.14.5.
>>> We have a number of scheduled tasks and I want to preserve
>>> those along>>> with other items in the queue. I tried moving the files in kahadb
>>> to the>>> new server (after shutting down both servers), but have run into
>>> multiple errors which prevent the new server from starting.
>>> I am wondering what the appropriate way to migrate storage is.
>>> Should I>>> attempt read all of the messages from the old queue and write
>>> them into>>> the new queue? Has anyone else done this before? It seems like this>>> would be a useful utility.
>>>
>>> Thanks,
>>>
>>> Chris
>>>
>>
Re: Migrating storage from 5.10.0 to 5.14.5
Posted by Tim Bain <tb...@alumni.duke.edu>.
What Chris described is how to load the existing messages into your new
broker. That has always seemed clunky to me because there is no way to
store messages in multiple OpenWire formats (which means your new broker's
store is stuck forever on the old broker's OpenWire version), so the option
that always sounded better to me is to stand up the new broker empty and
then attach the old broker in a network of brokers configuration. Clients
would connect to the new broker, and messages would flow from the old
broker to the new one and then to the consumers.
On May 24, 2017 2:45 PM, "Christopher Shannon" <
christopher.l.shannon@gmail.com> wrote:
> What kind of errors are you getting? If it's related to data marshalling
> then it could be because there was an Openwire upgrade (Openwire is the
> format the messages are stored in) done in the KahaDB store between those
> versions which could be causing your issue. I think there was also some
> scheduler work done so not sure off the top of my head if that is the
> problem without seeing the stack trace.
>
> If you are simply copying over the journal files and trying to rebuild the
> KahaDB index then it will fail as it will pick Openwire version 11 instead
> of 6. You set the version manually using the storeOpenWireVersion
> property. See http://activemq.apache.org/kahadb.html. If the index file
> still exists (db.data) then KahaDB should be able to detect that it is an
> old store and fall back automatically to the write openwire version.
>
>
>
> On Fri, May 19, 2017 at 2:32 PM, <ch...@baus.net> wrote:
>
> > Good Day,
> >
> > I've been working on migrating servers and upgrading from 5.10.0
> > to 5.14.5.
> > We have a number of scheduled tasks and I want to preserve those along
> > with other items in the queue. I tried moving the files in kahadb to the
> > new server (after shutting down both servers), but have run into
> > multiple errors which prevent the new server from starting.
> > I am wondering what the appropriate way to migrate storage is. Should I
> > attempt read all of the messages from the old queue and write them into
> > the new queue? Has anyone else done this before? It seems like this
> > would be a useful utility.
> >
> > Thanks,
> >
> > Chris
> >
>
Re: Migrating storage from 5.10.0 to 5.14.5
Posted by Christopher Shannon <ch...@gmail.com>.
What kind of errors are you getting? If it's related to data marshalling
then it could be because there was an Openwire upgrade (Openwire is the
format the messages are stored in) done in the KahaDB store between those
versions which could be causing your issue. I think there was also some
scheduler work done so not sure off the top of my head if that is the
problem without seeing the stack trace.
If you are simply copying over the journal files and trying to rebuild the
KahaDB index then it will fail as it will pick Openwire version 11 instead
of 6. You set the version manually using the storeOpenWireVersion
property. See http://activemq.apache.org/kahadb.html. If the index file
still exists (db.data) then KahaDB should be able to detect that it is an
old store and fall back automatically to the write openwire version.
On Fri, May 19, 2017 at 2:32 PM, <ch...@baus.net> wrote:
> Good Day,
>
> I've been working on migrating servers and upgrading from 5.10.0
> to 5.14.5.
> We have a number of scheduled tasks and I want to preserve those along
> with other items in the queue. I tried moving the files in kahadb to the
> new server (after shutting down both servers), but have run into
> multiple errors which prevent the new server from starting.
> I am wondering what the appropriate way to migrate storage is. Should I
> attempt read all of the messages from the old queue and write them into
> the new queue? Has anyone else done this before? It seems like this
> would be a useful utility.
>
> Thanks,
>
> Chris
>