You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Roland Thomas Lichti <ro...@telefonica.de> on 2010/03/17 13:43:41 UTC
Problem reading kahaPersistence data store
Hi,
after a broker shutdown I get IOExceptions when starting again. The
broker tries to ack a message but since the broker restarted the session
is gone. We use a kahaPersistenceAdapter (after having to much problems
with kahaDB). It is no solution to switch to another store, since we
need the messages inside the defective store. Is there any way to get
around the following exception?
We basically need only the data from the queues. If we get them in text
files, that's ok. We need name of destination, the properties and the
TextMessage inside the body. Since we only use TextMessages we don't
have to care about the different payload types.
2010-03-17 11:37:09,954 WARN
org.apache.activemq.broker.region.AbstractRegion - Ack for non existent
subscription, ack:MessageAck {commandId = 145, responseRequired = false, ack
Type = 2, consumerId = ID:solz-1556-54374-1268742357999-0:121:-1:2,
firstMessageId = ID:solz-1555-53110-1268744150357-0:86:49:1:1,
lastMessageId = ID:solz-1555-53110-126874415035
7-0:86:49:1:1, destination =
queue://A049_jms_q_intern_voiceJmsResponder, transactionId =
XID:131075:312d616537653932373a643161363a34623966373861363a3765353765:616537653932373a64
3161363a34623966373861363a3765353766, messageCount = 1}
2010-03-17 11:37:09,959 ERROR org.apache.activemq.broker.BrokerService -
Failed to start ActiveMQ JMS Message Broker. Reason:
java.io.IOException: Recovery Failed: The subscripti
on does not exist: ID:solz-1556-54374-1268742357999-0:121:-1:2
java.io.IOException: Recovery Failed: The subscription does not exist:
ID:solz-1556-54374-1268742357999-0:121:-1:2
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:27)
at
org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:110)
at
org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
at
org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
at
org.apache.activemq.broker.BrokerService$3.start(BrokerService.java:1668)
at
org.apache.activemq.broker.BrokerService.start(BrokerService.java:452)
at
org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:56)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1369)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(Native Method)
[...]
bye,
Roland
--
Roland Thomas Lichti
IT Application Manager
Telefónica o2 Germany GmbH & Co. OHG
Hülshorstweg 30, 33415 Verl
t: +49 5246 80 1121, f: +49 5246 80 2121
m: +49 160 98949570
roland.lichti@telefonica.de
http://www.telefonica.de
Bitte finden Sie hier die
handelsrechtlichen Pflichtangaben: http://www.telefonica.de/pflichtangaben.html
Re: Problem reading kahaPersistence data store
Posted by Roland Thomas Lichti <ro...@telefonica.de>.
Hi Gary,
Gary Tully schrieb:
> odd, that stack trace does not match with the 5.3.0 source.
That's funny, since I packaged the 5.3.0 into Sun Packages and used
kahaDB which hasn't been there before the 5.3.0 release (I never use
snapshots).
> Can you verify that it is
> 5.3.0 and that this installation has no duplicate jars or anything odd like
> that.
>
I'll check that, but I just got that error:
2010-03-19 10:08:53,441 INFO
org.apache.activemq.broker.TransportConnector - Connector stomp Stopped
2010-03-19 10:08:54,083 ERROR
org.apache.activemq.store.amq.AMQPersistenceAdapter - Could not stop
service: AMQPersistenceAdapter(/data/apps/TDEamqbro/data). Reason:
java.lang.Nu
llPointerException
java.lang.NullPointerException
at
org.apache.activemq.store.amq.AMQTxOperation.writeExternal(AMQTxOperation.java:113)
at
org.apache.activemq.store.kahadaptor.AMQTxMarshaller.writePayload(AMQTxMarshaller.java:60)
at
org.apache.activemq.store.kahadaptor.AMQTxMarshaller.writePayload(AMQTxMarshaller.java:34)
at
org.apache.activemq.kaha.impl.data.SyncDataFileWriter.storeItem(SyncDataFileWriter.java:58)
at
org.apache.activemq.kaha.impl.data.DataManagerImpl.storeDataItem(DataManagerImpl.java:152)
at
org.apache.activemq.kaha.impl.container.MapContainerImpl.write(MapContainerImpl.java:532)
at
org.apache.activemq.kaha.impl.container.MapContainerImpl.put(MapContainerImpl.java:317)
at
org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.savePreparedState(KahaReferenceStoreAdapter.java:306)
at
org.apache.activemq.store.amq.AMQPersistenceAdapter.stop(AMQPersistenceAdapter.java:324)
at
org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41)
at
org.apache.activemq.broker.BrokerService.stop(BrokerService.java:519)
at
org.apache.activemq.console.command.StartCommand$1.run(StartCommand.java:135)
2010-03-19 10:08:54,090 INFO org.apache.activemq.broker.BrokerService -
ActiveMQ JMS Message Broker (solr-2073,
ID:solr-2073-43574-1268986625273-0:0) stopped
I don't like reading NullPointerException when writing persistence
stores ...
> So the root cause seems to be connections not shutting down which forces you
> to kill the broker, which leads to this state.
>
That seemed to be the problem but here the connection problem wasn't
there and the store failed.
bye,
Roland
--
Roland Thomas Lichti
IT Application Manager
Telefónica o2 Germany GmbH & Co. OHG
Hülshorstweg 30, 33415 Verl
t: +49 5246 80 1121, f: +49 5246 80 2121
m: +49 160 98949570
roland.lichti@telefonica.de
http://www.telefonica.de
Bitte finden Sie hier die
handelsrechtlichen Pflichtangaben: http://www.telefonica.de/pflichtangaben.html
Re: Problem reading kahaPersistence data store
Posted by Gary Tully <ga...@gmail.com>.
odd, that stack trace does not match with the 5.3.0 source. I missing
subscription should be ignored during recovery. Can you verify that it is
5.3.0 and that this installation has no duplicate jars or anything odd like
that.
So the root cause seems to be connections not shutting down which forces you
to kill the broker, which leads to this state.
If you need to kill the broker again for this reason, can you first create a
thread dump so we can determine the cause for the connection that is
hanging/taking a long time to shutdown.
On 18 March 2010 15:38, Roland Thomas Lichti <ro...@telefonica.de>wrote:
> Gary Tully schrieb:
> > what version are you using? this looks like
> > http://issues.apache.org/activemq/browse/AMQ-1712 but that is not
> consistent
> > with your reference to KahaDB.
> >
> Sorry. The broker is 5.3.0. We had no problems with our stores on 5.2.0
> but there we had problems with hanging connections. These hanging
> connections went away when we switched to 5.3.0. But now we loose
> messages due to corrupt stores.
>
> Today we lost another store (this time amqPersistence, to wich we
> switched after we lost the kahaPeristence store my first message told
> from).
>
> This time we had to kill the broker since there were connections hanging
> ("connection /xx.yy.zz.aa:12345 taking a long time ..."). I checked with
> netstat and there were no connections with that IP and port number -
> seems to be a hickup in AMQ. But without closing these connections the
> store does not get flushed. And since we needed a working broker we were
> forced to kill that broker. And we lost again a store.
>
> At the moment my decision to use AMQ is hardly challanged by my boss and
> his peers ...
>
> bye,
> Roland
>
> --
> Roland Thomas Lichti
> IT Application Manager
>
> Telefónica o2 Germany GmbH & Co. OHG
> Hülshorstweg 30, 33415 Verl
> t: +49 5246 80 1121, f: +49 5246 80 2121
> m: +49 160 98949570
> roland.lichti@telefonica.de
> http://www.telefonica.de
>
> Bitte finden Sie hier die
> handelsrechtlichen Pflichtangaben:
> http://www.telefonica.de/pflichtangaben.html
>
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Problem reading kahaPersistence data store
Posted by Roland Thomas Lichti <ro...@telefonica.de>.
Gary Tully schrieb:
> what version are you using? this looks like
> http://issues.apache.org/activemq/browse/AMQ-1712 but that is not consistent
> with your reference to KahaDB.
>
Sorry. The broker is 5.3.0. We had no problems with our stores on 5.2.0
but there we had problems with hanging connections. These hanging
connections went away when we switched to 5.3.0. But now we loose
messages due to corrupt stores.
Today we lost another store (this time amqPersistence, to wich we
switched after we lost the kahaPeristence store my first message told from).
This time we had to kill the broker since there were connections hanging
("connection /xx.yy.zz.aa:12345 taking a long time ..."). I checked with
netstat and there were no connections with that IP and port number -
seems to be a hickup in AMQ. But without closing these connections the
store does not get flushed. And since we needed a working broker we were
forced to kill that broker. And we lost again a store.
At the moment my decision to use AMQ is hardly challanged by my boss and
his peers ...
bye,
Roland
--
Roland Thomas Lichti
IT Application Manager
Telefónica o2 Germany GmbH & Co. OHG
Hülshorstweg 30, 33415 Verl
t: +49 5246 80 1121, f: +49 5246 80 2121
m: +49 160 98949570
roland.lichti@telefonica.de
http://www.telefonica.de
Bitte finden Sie hier die
handelsrechtlichen Pflichtangaben: http://www.telefonica.de/pflichtangaben.html
Re: Problem reading kahaPersistence data store
Posted by Gary Tully <ga...@gmail.com>.
what version are you using? this looks like
http://issues.apache.org/activemq/browse/AMQ-1712 but that is not consistent
with your reference to KahaDB.
On 17 March 2010 12:43, Roland Thomas Lichti <ro...@telefonica.de>wrote:
> Hi,
>
> after a broker shutdown I get IOExceptions when starting again. The
> broker tries to ack a message but since the broker restarted the session
> is gone. We use a kahaPersistenceAdapter (after having to much problems
> with kahaDB). It is no solution to switch to another store, since we
> need the messages inside the defective store. Is there any way to get
> around the following exception?
>
> We basically need only the data from the queues. If we get them in text
> files, that's ok. We need name of destination, the properties and the
> TextMessage inside the body. Since we only use TextMessages we don't
> have to care about the different payload types.
>
>
> 2010-03-17 11:37:09,954 WARN
> org.apache.activemq.broker.region.AbstractRegion - Ack for non existent
> subscription, ack:MessageAck {commandId = 145, responseRequired = false,
> ack
> Type = 2, consumerId = ID:solz-1556-54374-1268742357999-0:121:-1:2,
> firstMessageId = ID:solz-1555-53110-1268744150357-0:86:49:1:1,
> lastMessageId = ID:solz-1555-53110-126874415035
> 7-0:86:49:1:1, destination =
> queue://A049_jms_q_intern_voiceJmsResponder, transactionId =
>
> XID:131075:312d616537653932373a643161363a34623966373861363a3765353765:616537653932373a64
> 3161363a34623966373861363a3765353766, messageCount = 1}
> 2010-03-17 11:37:09,959 ERROR org.apache.activemq.broker.BrokerService -
> Failed to start ActiveMQ JMS Message Broker. Reason:
> java.io.IOException: Recovery Failed: The subscripti
> on does not exist: ID:solz-1556-54374-1268742357999-0:121:-1:2
> java.io.IOException: Recovery Failed: The subscription does not exist:
> ID:solz-1556-54374-1268742357999-0:121:-1:2
> at
>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:27)
> at
>
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:110)
> at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
> at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
> at
> org.apache.activemq.broker.BrokerService$3.start(BrokerService.java:1668)
> at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:452)
> at
>
> org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:56)
> at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1369)
> at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)
> at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
> at
>
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
> at java.security.AccessController.doPrivileged(Native Method)
> [...]
>
> bye,
> Roland
>
> --
> Roland Thomas Lichti
> IT Application Manager
>
> Telefónica o2 Germany GmbH & Co. OHG
> Hülshorstweg 30, 33415 Verl
> t: +49 5246 80 1121, f: +49 5246 80 2121
> m: +49 160 98949570
> roland.lichti@telefonica.de
> http://www.telefonica.de
>
> Bitte finden Sie hier die
> handelsrechtlichen Pflichtangaben:
> http://www.telefonica.de/pflichtangaben.html
>
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com