You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Arjen van der Meijden <ac...@tweakers.net> on 2010/08/14 20:05:27 UTC
Some stomp-messages are received with an empty text
Hi List,
I have a fairly simple set-up with a single Activemq (currently 5.3.2,
but the behavior existed with previous versions).
On that AMQ we have a few queue's and topics, where the topics are
filled using PHP producers with Stomp and four Java subscribers with the
normal openwire protocol (via a Spring jms listener container). The PHP
producers are created on demand and send a single, tiny text message
when some data needs to be adjusted in the Java subscribers.
The problem is that all four consumers sometimes receive a message
without such a text body, where it should have contained one.
For instance, we have a hourly cronjob that sends an update after its
done ('views 0'). This cronjob is executed on the same machine, with the
same php-scripts etc. Today, the message was empty at 7u, 10u, 11u, 15u,
17u and 19u, although yesterday only at 6u, 10u and 20u.
So for now my question would be: how to debug this? I can trivially
reproduce the situation by just sending a few bogus-messages some of
those will fail. I can't see any difference in the php-side between
failed and successful messages. But the server is used quite heavily
with messages over the queues, so just changing the log level to debug
won't be a good start (unless only a specific topic could be in
debugmode?). I could separate the queues from the topics for now, but
than I have no guarantee the same situation occurs.
I've no idea where to look, but I'm actually suspecting ActiveMQ for now
as I can't really believe my php-scripts would fail only some of the
time (normally it fails or it works).
Any idea where I should look?
Best regards,
Arjen
My configuration is rather trivial, its the default config without
systemusage and destinationpolicy. With the stomp and openwire
transporters like so:
<transportConnector name="stomp"
uri="stomp://0.0.0.0:61613?transport.closeAsync=false"/>
When logging the messages in the subscribers, it looks like this for a
correct one:
ActiveMQTextMessage {commandId = 3, responseRequired = false, messageId
= ID:argus-38958-1279890714168-2:66257231:-1:1:1, originalDestination =
null, originalTransactionId = null, producerId =
ID:argus-38958-1279890714168-2:66257231:-1:1, destination =
topic://pw_cache_products, transactionId = null, expiration = 0,
timestamp = 1281801902987, arrival = 0, brokerInTime = 1281801902987,
brokerOutTime = 1281801902988, correlationId = null, replyTo = null,
persistent = false, type = null, priority = 0, groupID = null,
groupSequence = 0, targetConsumerId = null, compressed = false, userID =
null, content = null, marshalledProperties = null, dataStructure = null,
redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties =
true, readOnlyBody = true, droppable = false, text = views 0}
And like this for an incorrect one:
ActiveMQTextMessage {commandId = 3, responseRequired = false, messageId
= ID:argus-38958-1279890714168-2:66384399:-1:1:1, originalDestination =
null, originalTransactionId = null, producerId =
ID:argus-38958-1279890714168-2:66384399:-1:1, destination =
topic://pw_cache_products, transactionId = null, expiration = 0,
timestamp = 1281805503252, arrival = 0, brokerInTime = 1281805503252,
brokerOutTime = 1281805503252, correlationId = null, replyTo = null,
persistent = false, type = null, priority = 0, groupID = null,
groupSequence = 0, targetConsumerId = null, compressed = false, userID =
null, content = null, marshalledProperties = null, dataStructure = null,
redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties =
true, readOnlyBody = true, droppable = false, text = null}
I.e. text = null vs text = views 0
Re: Some stomp-messages are received with an empty text
Posted by Arjen van der Meijden <ac...@tweakers.net>.
I doubt it, it already are text messages and both the correctly
delivered and empty messages are TextMessage according to the client.
Arjen
On 25-8-2010 7:54, Antony Bowesman wrote:
> I vaguely remember seeing something similar, not sure if it was over
> XMPP or Stomp, but I was sending BytesMessages and the client got
> nothing. I changed it to send TextMessage and all was OK.
>
> Could this be your problem?
>
> On 25/08/2010 3:49 PM, Arjen van der Meijden wrote:
>> Noone? I've done some follow-up research, but still have no idea which
>> part of ActiveMQ fails. It turns out that the incoming stomp-messages
>> are all good, so they're received correctly. And I've also noticed that,
>> by looking at the tcp-traffic, the outgoing messages indeed don't have
>> the body so now and then.
>>
>> So the text-message body disappears inside ActiveMQ. It looks to me as
>> if some internal resource is shared in a place it shouldn't have been
>> shared and is reset to null before others are done with it. But I
>> couldn't find such a resource in the o.a...transport.stomp-package. Then
>> again I don't know the code that well, so that doesn't mean much.
>>
>> And while I can easily reproduce (on that specific AMQ-instance) the
>> problem with any topic, I can't reproduce it on another AMQ. So I'm
>> guessing the load inhibited on our production-instance has something to
>> do with it as well (which supports the idea of some shared resource that
>> shouldn't be shared amongst threads).
>>
>> I've already filed a JIRA-issue, but would really like to figure this
>> out and find a solution.
>> https://issues.apache.org/activemq/browse/AMQ-2871
>>
>> Any ideas how to proceed from here?
>>
>> Best regards,
>>
>> Arjen
>
Re: Some stomp-messages are received with an empty text
Posted by Antony Bowesman <ad...@thorntothehorn.org>.
I vaguely remember seeing something similar, not sure if it was over XMPP or
Stomp, but I was sending BytesMessages and the client got nothing. I changed it
to send TextMessage and all was OK.
Could this be your problem?
On 25/08/2010 3:49 PM, Arjen van der Meijden wrote:
> Noone? I've done some follow-up research, but still have no idea which
> part of ActiveMQ fails. It turns out that the incoming stomp-messages
> are all good, so they're received correctly. And I've also noticed that,
> by looking at the tcp-traffic, the outgoing messages indeed don't have
> the body so now and then.
>
> So the text-message body disappears inside ActiveMQ. It looks to me as
> if some internal resource is shared in a place it shouldn't have been
> shared and is reset to null before others are done with it. But I
> couldn't find such a resource in the o.a...transport.stomp-package. Then
> again I don't know the code that well, so that doesn't mean much.
>
> And while I can easily reproduce (on that specific AMQ-instance) the
> problem with any topic, I can't reproduce it on another AMQ. So I'm
> guessing the load inhibited on our production-instance has something to
> do with it as well (which supports the idea of some shared resource that
> shouldn't be shared amongst threads).
>
> I've already filed a JIRA-issue, but would really like to figure this
> out and find a solution.
> https://issues.apache.org/activemq/browse/AMQ-2871
>
> Any ideas how to proceed from here?
>
> Best regards,
>
> Arjen
Re: Some stomp-messages are received with an empty text
Posted by Arjen van der Meijden <ac...@tweakers.net>.
Noone? I've done some follow-up research, but still have no idea which
part of ActiveMQ fails. It turns out that the incoming stomp-messages
are all good, so they're received correctly. And I've also noticed that,
by looking at the tcp-traffic, the outgoing messages indeed don't have
the body so now and then.
So the text-message body disappears inside ActiveMQ. It looks to me as
if some internal resource is shared in a place it shouldn't have been
shared and is reset to null before others are done with it. But I
couldn't find such a resource in the o.a...transport.stomp-package. Then
again I don't know the code that well, so that doesn't mean much.
And while I can easily reproduce (on that specific AMQ-instance) the
problem with any topic, I can't reproduce it on another AMQ. So I'm
guessing the load inhibited on our production-instance has something to
do with it as well (which supports the idea of some shared resource that
shouldn't be shared amongst threads).
I've already filed a JIRA-issue, but would really like to figure this
out and find a solution.
https://issues.apache.org/activemq/browse/AMQ-2871
Any ideas how to proceed from here?
Best regards,
Arjen
On 14-8-2010 20:05 Arjen van der Meijden wrote:
> Hi List,
>
> I have a fairly simple set-up with a single Activemq (currently 5.3.2,
> but the behavior existed with previous versions).
>
> On that AMQ we have a few queue's and topics, where the topics are
> filled using PHP producers with Stomp and four Java subscribers with the
> normal openwire protocol (via a Spring jms listener container). The PHP
> producers are created on demand and send a single, tiny text message
> when some data needs to be adjusted in the Java subscribers.
>
> The problem is that all four consumers sometimes receive a message
> without such a text body, where it should have contained one.
>
> For instance, we have a hourly cronjob that sends an update after its
> done ('views 0'). This cronjob is executed on the same machine, with the
> same php-scripts etc. Today, the message was empty at 7u, 10u, 11u, 15u,
> 17u and 19u, although yesterday only at 6u, 10u and 20u.
>
> So for now my question would be: how to debug this? I can trivially
> reproduce the situation by just sending a few bogus-messages some of
> those will fail. I can't see any difference in the php-side between
> failed and successful messages. But the server is used quite heavily
> with messages over the queues, so just changing the log level to debug
> won't be a good start (unless only a specific topic could be in
> debugmode?). I could separate the queues from the topics for now, but
> than I have no guarantee the same situation occurs.
>
> I've no idea where to look, but I'm actually suspecting ActiveMQ for now
> as I can't really believe my php-scripts would fail only some of the
> time (normally it fails or it works).
>
> Any idea where I should look?
>
> Best regards,
>
> Arjen
>
> My configuration is rather trivial, its the default config without
> systemusage and destinationpolicy. With the stomp and openwire
> transporters like so:
> <transportConnector name="stomp"
> uri="stomp://0.0.0.0:61613?transport.closeAsync=false"/>
>
>
> When logging the messages in the subscribers, it looks like this for a
> correct one:
> ActiveMQTextMessage {commandId = 3, responseRequired = false, messageId
> = ID:argus-38958-1279890714168-2:66257231:-1:1:1, originalDestination =
> null, originalTransactionId = null, producerId =
> ID:argus-38958-1279890714168-2:66257231:-1:1, destination =
> topic://pw_cache_products, transactionId = null, expiration = 0,
> timestamp = 1281801902987, arrival = 0, brokerInTime = 1281801902987,
> brokerOutTime = 1281801902988, correlationId = null, replyTo = null,
> persistent = false, type = null, priority = 0, groupID = null,
> groupSequence = 0, targetConsumerId = null, compressed = false, userID =
> null, content = null, marshalledProperties = null, dataStructure = null,
> redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties =
> true, readOnlyBody = true, droppable = false, text = views 0}
>
> And like this for an incorrect one:
> ActiveMQTextMessage {commandId = 3, responseRequired = false, messageId
> = ID:argus-38958-1279890714168-2:66384399:-1:1:1, originalDestination =
> null, originalTransactionId = null, producerId =
> ID:argus-38958-1279890714168-2:66384399:-1:1, destination =
> topic://pw_cache_products, transactionId = null, expiration = 0,
> timestamp = 1281805503252, arrival = 0, brokerInTime = 1281805503252,
> brokerOutTime = 1281805503252, correlationId = null, replyTo = null,
> persistent = false, type = null, priority = 0, groupID = null,
> groupSequence = 0, targetConsumerId = null, compressed = false, userID =
> null, content = null, marshalledProperties = null, dataStructure = null,
> redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties =
> true, readOnlyBody = true, droppable = false, text = null}
>
> I.e. text = null vs text = views 0
>
>