You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Martin C." <ma...@gmx.at> on 2011/03/11 09:27:19 UTC
EOFException when accessing message's properties
Hi,
I am experiencing a corrupt message (see stacktrace below) which fails
when attempting to access the message's properties. We had this on two
occurances until now, both on a 5.3.2 broker. The issue happens
persists on 5.4.2, if we start it with the KahaDB datastore of the
5.3.2 instance. The message finally ended in a DLQ, but it also cannot
be processed from there.
The issue seems to only happen when deserializing the message (either
via network or from KahaDB on startup), as the message was viewable in
the webinterface of the broker before a restart, but afterwards, the
same exception occures when trying to browse the queue containing the
message in the web-interface.
Unfortunately I do not know a way to reproduce this, as this happens
only once every several-million messages. I also unfortunately cannot
provide the corrupt datastore due to sensitive data being included.
Do you have any idea, if/how I can recover the message somehow?
javax.jms.JMSException: java.io.EOFException
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at org.apache.activemq.filter.PropertyExpression.evaluate(PropertyExpression.java:199)
at org.apache.activemq.command.ActiveMQMessage.getObjectProperty(ActiveMQMessage.java:462)
at org.apache.activemq.command.ActiveMQMessage.getStringProperty(ActiveMQMessage.java:550)
... snip: some myapp.internal method calls ...
at myapp.MessageHandler.onMessage(MessageHandler.java:101)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:534)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:241)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:977)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
at java.io.DataInputStream.readUTF(DataInputStream.java:572)
at java.io.DataInputStream.readUTF(DataInputStream.java:547)
at org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:92)
at org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:73)
at org.apache.activemq.command.Message.unmarsallProperties(Message.java:189)
at org.apache.activemq.command.Message.getProperty(Message.java:151)
at org.apache.activemq.filter.PropertyExpression.evaluate(PropertyExpression.java:197)
... 15 more
Thanks in advance,
Martin
Re: EOFException when accessing message's properties
Posted by "Martin C." <ma...@gmx.at>.
Hi
for further information if anybody encounters the same problem:
The issue with the corrupted message headers really seems to be
closely related to the persistence, as I got completely rid of the
corrupted messages by disabling parallel dispatching and persistence
in KahaDB by using:
<kahaDB directory="${activemq.base}/data/kahadb"
checkForCorruptJournalFiles="true" checksumJournalFiles="true"
concurrentStoreAndDispatchTopics="false"
concurrentStoreAndDispatchQueues="false"
/>
I am not sure if this really resolves the issue completely, but at
least it didn't happen again for several tenth of millions of messages
until now.
Best regards,
Martin
On Thu, Apr 7, 2011 at 3:15 PM, Martin C. <ma...@gmx.at> wrote:
> Hi,
>
> I investigated the issue further.
>
> In the corresponding message, the "marshalledProperties" seems to be
> corrupt. The length header indicates 4 properties being present, but
> only two marshalled properties are in the "marshalledProperties"
> array. This is the reason for the EOFException during parsing.
>
> Also interestingly, this only happened to persistent messages. On the
> broker, they seem to be valid (they can be browsed in the web
> interface). Until restart of the broker, when they are loaded from
> persistent storage, then I cannot browse for them in the queue due to
> the EOFException happening on the web-interface as well.
>
> Should I search for the issue on the broker, the consumer or the
> producer (my guess being the broker)? Any hints?
>
> Thanks in advance,
> Martin
>
> On Thu, Apr 7, 2011 at 12:19 AM, Martin C. <ma...@gmx.at> wrote:
>> Hi,
>>
>> as a follow-up, I'm currently seeing a very highly elevated error rate
>> on a deployment with 5.4.2. The very strange issue here is that if I
>> move the message out of the DLQ into the original queue, after some
>> attempts it gets consumed without problems.
>>
>> Anybody got any ideas? Could this have to do with prefetch limit being
>> set to 1 and XA transactions (I read about issues with prefetch and
>> spring on the homepage, but this indicates that one should be a valid
>> limit).
>>
>> Thanks in advance for any help!
>>
>> Best regards,
>> Martin
>>
>> ---------- Forwarded message ----------
>> From: Martin C. <ma...@gmx.at>
>> Date: Fri, Mar 11, 2011 at 9:27 AM
>> Subject: EOFException when accessing message's properties
>> To: users <us...@activemq.apache.org>
>>
>>
>> Hi,
>>
>> I am experiencing a corrupt message (see stacktrace below) which fails
>> when attempting to access the message's properties. We had this on two
>> occurances until now, both on a 5.3.2 broker. The issue happens
>> persists on 5.4.2, if we start it with the KahaDB datastore of the
>> 5.3.2 instance. The message finally ended in a DLQ, but it also cannot
>> be processed from there.
>>
>> The issue seems to only happen when deserializing the message (either
>> via network or from KahaDB on startup), as the message was viewable in
>> the webinterface of the broker before a restart, but afterwards, the
>> same exception occures when trying to browse the queue containing the
>> message in the web-interface.
>> Unfortunately I do not know a way to reproduce this, as this happens
>> only once every several-million messages. I also unfortunately cannot
>> provide the corrupt datastore due to sensitive data being included.
>>
>> Do you have any idea, if/how I can recover the message somehow?
>>
>> javax.jms.JMSException: java.io.EOFException
>> at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
>> at org.apache.activemq.filter.PropertyExpression.evaluate(PropertyExpression.java:199)
>> at org.apache.activemq.command.ActiveMQMessage.getObjectProperty(ActiveMQMessage.java:462)
>> at org.apache.activemq.command.ActiveMQMessage.getStringProperty(ActiveMQMessage.java:550)
>> ... snip: some myapp.internal method calls ...
>> at myapp.MessageHandler.onMessage(MessageHandler.java:101)
>> at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:534)
>> at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
>> at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
>> at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:241)
>> at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:977)
>> at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.io.EOFException
>> at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
>> at java.io.DataInputStream.readUTF(DataInputStream.java:572)
>> at java.io.DataInputStream.readUTF(DataInputStream.java:547)
>> at org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:92)
>> at org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:73)
>> at org.apache.activemq.command.Message.unmarsallProperties(Message.java:189)
>> at org.apache.activemq.command.Message.getProperty(Message.java:151)
>> at org.apache.activemq.filter.PropertyExpression.evaluate(PropertyExpression.java:197)
>> ... 15 more
>>
>>
>> Thanks in advance,
>> Martin
>>
>
Re: EOFException when accessing message's properties
Posted by "Martin C." <ma...@gmx.at>.
Hi,
I investigated the issue further.
In the corresponding message, the "marshalledProperties" seems to be
corrupt. The length header indicates 4 properties being present, but
only two marshalled properties are in the "marshalledProperties"
array. This is the reason for the EOFException during parsing.
Also interestingly, this only happened to persistent messages. On the
broker, they seem to be valid (they can be browsed in the web
interface). Until restart of the broker, when they are loaded from
persistent storage, then I cannot browse for them in the queue due to
the EOFException happening on the web-interface as well.
Should I search for the issue on the broker, the consumer or the
producer (my guess being the broker)? Any hints?
Thanks in advance,
Martin
On Thu, Apr 7, 2011 at 12:19 AM, Martin C. <ma...@gmx.at> wrote:
> Hi,
>
> as a follow-up, I'm currently seeing a very highly elevated error rate
> on a deployment with 5.4.2. The very strange issue here is that if I
> move the message out of the DLQ into the original queue, after some
> attempts it gets consumed without problems.
>
> Anybody got any ideas? Could this have to do with prefetch limit being
> set to 1 and XA transactions (I read about issues with prefetch and
> spring on the homepage, but this indicates that one should be a valid
> limit).
>
> Thanks in advance for any help!
>
> Best regards,
> Martin
>
> ---------- Forwarded message ----------
> From: Martin C. <ma...@gmx.at>
> Date: Fri, Mar 11, 2011 at 9:27 AM
> Subject: EOFException when accessing message's properties
> To: users <us...@activemq.apache.org>
>
>
> Hi,
>
> I am experiencing a corrupt message (see stacktrace below) which fails
> when attempting to access the message's properties. We had this on two
> occurances until now, both on a 5.3.2 broker. The issue happens
> persists on 5.4.2, if we start it with the KahaDB datastore of the
> 5.3.2 instance. The message finally ended in a DLQ, but it also cannot
> be processed from there.
>
> The issue seems to only happen when deserializing the message (either
> via network or from KahaDB on startup), as the message was viewable in
> the webinterface of the broker before a restart, but afterwards, the
> same exception occures when trying to browse the queue containing the
> message in the web-interface.
> Unfortunately I do not know a way to reproduce this, as this happens
> only once every several-million messages. I also unfortunately cannot
> provide the corrupt datastore due to sensitive data being included.
>
> Do you have any idea, if/how I can recover the message somehow?
>
> javax.jms.JMSException: java.io.EOFException
> at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
> at org.apache.activemq.filter.PropertyExpression.evaluate(PropertyExpression.java:199)
> at org.apache.activemq.command.ActiveMQMessage.getObjectProperty(ActiveMQMessage.java:462)
> at org.apache.activemq.command.ActiveMQMessage.getStringProperty(ActiveMQMessage.java:550)
> ... snip: some myapp.internal method calls ...
> at myapp.MessageHandler.onMessage(MessageHandler.java:101)
> at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:534)
> at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
> at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
> at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:241)
> at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:977)
> at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
> at java.io.DataInputStream.readUTF(DataInputStream.java:572)
> at java.io.DataInputStream.readUTF(DataInputStream.java:547)
> at org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:92)
> at org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:73)
> at org.apache.activemq.command.Message.unmarsallProperties(Message.java:189)
> at org.apache.activemq.command.Message.getProperty(Message.java:151)
> at org.apache.activemq.filter.PropertyExpression.evaluate(PropertyExpression.java:197)
> ... 15 more
>
>
> Thanks in advance,
> Martin
>
Re: EOFException when accessing message's properties
Posted by "Martin C." <ma...@gmx.at>.
Hi,
as a follow-up, I'm currently seeing a very highly elevated error rate
on a deployment with 5.4.2. The very strange issue here is that if I
move the message out of the DLQ into the original queue, after some
attempts it gets consumed without problems.
Anybody got any ideas? Could this have to do with prefetch limit being
set to 1 and XA transactions (I read about issues with prefetch and
spring on the homepage, but this indicates that one should be a valid
limit).
Thanks in advance for any help!
Best regards,
Martin
---------- Forwarded message ----------
From: Martin C. <ma...@gmx.at>
Date: Fri, Mar 11, 2011 at 9:27 AM
Subject: EOFException when accessing message's properties
To: users <us...@activemq.apache.org>
Hi,
I am experiencing a corrupt message (see stacktrace below) which fails
when attempting to access the message's properties. We had this on two
occurances until now, both on a 5.3.2 broker. The issue happens
persists on 5.4.2, if we start it with the KahaDB datastore of the
5.3.2 instance. The message finally ended in a DLQ, but it also cannot
be processed from there.
The issue seems to only happen when deserializing the message (either
via network or from KahaDB on startup), as the message was viewable in
the webinterface of the broker before a restart, but afterwards, the
same exception occures when trying to browse the queue containing the
message in the web-interface.
Unfortunately I do not know a way to reproduce this, as this happens
only once every several-million messages. I also unfortunately cannot
provide the corrupt datastore due to sensitive data being included.
Do you have any idea, if/how I can recover the message somehow?
javax.jms.JMSException: java.io.EOFException
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at org.apache.activemq.filter.PropertyExpression.evaluate(PropertyExpression.java:199)
at org.apache.activemq.command.ActiveMQMessage.getObjectProperty(ActiveMQMessage.java:462)
at org.apache.activemq.command.ActiveMQMessage.getStringProperty(ActiveMQMessage.java:550)
... snip: some myapp.internal method calls ...
at myapp.MessageHandler.onMessage(MessageHandler.java:101)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:534)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:241)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:977)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:876)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
at java.io.DataInputStream.readUTF(DataInputStream.java:572)
at java.io.DataInputStream.readUTF(DataInputStream.java:547)
at org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:92)
at org.apache.activemq.util.MarshallingSupport.unmarshalPrimitiveMap(MarshallingSupport.java:73)
at org.apache.activemq.command.Message.unmarsallProperties(Message.java:189)
at org.apache.activemq.command.Message.getProperty(Message.java:151)
at org.apache.activemq.filter.PropertyExpression.evaluate(PropertyExpression.java:197)
... 15 more
Thanks in advance,
Martin