You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by ajls77 <aj...@gmail.com> on 2007/11/27 15:07:38 UTC

Messages appear to get lost

I am sending messages to an ActiveMQ broker from an Java5 ExecutorService
thread pool.  Concurrency requirements at the moment require at least 60
MessageProducers - these, along with a Session, live in a thread-safe FIFO
pipe. 

When concurrency hots up, it appears that some messages are being lost -
from the client's perspective it appears to have been delivered, but the
debugging output on the broker does not provide a receipt of the message
(the message delivery is set to PERSISTENT just to get the debugging as
there is no debugging information under NON_PERSISTENT).

To prove that the client thinks the message is delivered I recompiled
activemq-core-4.1.1.jar with extra debugging.  I traced the following
execution flow:

ActiveMQMessageProducer.send()
-> ActiveMQSession.send()
 -> TcpTransport.oneway()
   -> TcpBufferedOutputStream.flush()

The output of which is:

DEBUG: org.apache.activemq.ActiveMQSession - sending message with messageID:
ID:ukedfrog-3905-1196171536066-1:5:1:1:54 Thread:
Thread[pool-1-thread-493,5,main]
DEBUG: org.apache.activemq.ActiveMQSession - Sending message:
ActiveMQTextMessage {commandId = 0, responseRequired = false, messageId =
ID:ukedfrog-3905-1196171536066-1:5:1:1:54, originalDestination = null,
originalTransactionId = null, producerId =
ID:ukedfrog-3905-1196171536066-1:5:1:1, destination =
queue://urn:telelogic:ers:two, transactionId = null, expiration = 0,
timestamp = 1196171643911, arrival = 0, correlationId = null, replyTo =
queue://reply-192.168.68.151@urn:telelogic:ers:two, persistent = true, type
= null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId =
null, compressed = false, userID = null, content = null,
marshalledProperties = null, dataStructure = null, redeliveryCounter = 0,
size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true,
droppable = false, text = messageContent}
DEBUG org.apache.activemq.transport.tcp.TcpBufferedOutputStream calling
flush - messageID: ID:ukedfrog-3905-1196171536066-1:5:1:1:54 count: 116
OutputStream: java.net.SocketOutputStream@94124f
DEBUG org.apache.activemq.transport.tcp.TcpTransport flushing bytes -
messageID:ID:ukedfrog-3905-1196171536066-1:5:1:1:54 Thread:
Thread[pool-1-thread-493,5,main]

(The messageID is propagated to the activemq classes via a ThreadLocal)

My understanding is that after TcpBufferedOutputStream.flush() the TCP
packet is delegated to the underlying operating system and the message has
effectively been "sent".  However, there is no receipt of this message in
the broker, no matter how long we wait.  Is this message "lost" ? or is my
client implementation faulty.

I am using one Connection per JVM and used by each thread (with its own
MessageProducer & Session). After reading 
http://gnodet.blogspot.com/2006/12/over-past-weeks-i-spend-some-times-load.html
this  , I tried using a separate Connection per thread, but the problem
persisted.  This write-up concerns me  (that the broker processes all
messages on a given Connection sequentially)  as this would force the client
implementation to have multiple Connections which does not seem to be
mandated by the JMS spec.  


My ConnectionFactory is configured to use synchronous sending via
"connectionFactory.setUseAsyncSend(false);"

Could anybody explain why these message appear to be "lost"? It seems, from
client perspective, that the message has been delivered, but it never is.

Many thanks
-- 
View this message in context: http://www.nabble.com/Messages-appear-to-get-lost-tf4881925s2354.html#a13971307
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Solved: Messages appear to get lost

Posted by ajls77 <aj...@gmail.com>.
Hmmmm ... as suspected, a silly error in the client thread model was causing
two threads to use the same producer.  



ajls77 wrote:
> 
> Thanks for the response.  I have bumped by my usageManager's resource
> allocation to 512Mb and the problem still persists.  Funnily enough, it
> seems easier to replicate the problem with 10 threads producing every
> second than 250 per second.  So it cannot be an out and out
> concurrency-induced.  I am still quite suspicious that my thread model
> (ThreadPoolExecutor - and some LockSupport stuff) may be causing the
> problem.
> 
> To verify this, could you help clarify these assumptions ?
> 
> When TcpBufferedOutputStream.flush() is called, and useAsyncSend(false),
> the message, as far as the JVM is concerned, has "left the building".
> 
> and
> 
> If not, what part of ActiveMQ could affect the sending after this point.
> 
> 
> I am convinced that TcpBufferedOutputStream.flush() is executing correctly
> for a given message ID.   
> 
> Many thanks
> 
> P.S. I cannot, as of now use 5.x, but the message cursors will give a nice
> amount of control over dispatching.  
> 
> 
> James.Strachan wrote:
>> 
>> Maybe you're hitting the flow control causing some producers to be
>> blocked. See this FAQ entry...
>> http://activemq.apache.org/my-producer-blocks.html
>> 
>> You might wanna try using 5.x which has much better support for
>> dealing with flow control in different ways.
>> 
>> 
>> On 27/11/2007, ajls77 <aj...@gmail.com> wrote:
>>>
>>> I am sending messages to an ActiveMQ broker from an Java5
>>> ExecutorService
>>> thread pool.  Concurrency requirements at the moment require at least 60
>>> MessageProducers - these, along with a Session, live in a thread-safe
>>> FIFO
>>> pipe.
>>>
>>> When concurrency hots up, it appears that some messages are being lost -
>>> from the client's perspective it appears to have been delivered, but the
>>> debugging output on the broker does not provide a receipt of the message
>>> (the message delivery is set to PERSISTENT just to get the debugging as
>>> there is no debugging information under NON_PERSISTENT).
>>>
>>> To prove that the client thinks the message is delivered I recompiled
>>> activemq-core-4.1.1.jar with extra debugging.  I traced the following
>>> execution flow:
>>>
>>> ActiveMQMessageProducer.send()
>>> -> ActiveMQSession.send()
>>>  -> TcpTransport.oneway()
>>>    -> TcpBufferedOutputStream.flush()
>>>
>>> The output of which is:
>>>
>>> DEBUG: org.apache.activemq.ActiveMQSession - sending message with
>>> messageID:
>>> ID:ukedfrog-3905-1196171536066-1:5:1:1:54 Thread:
>>> Thread[pool-1-thread-493,5,main]
>>> DEBUG: org.apache.activemq.ActiveMQSession - Sending message:
>>> ActiveMQTextMessage {commandId = 0, responseRequired = false, messageId
>>> =
>>> ID:ukedfrog-3905-1196171536066-1:5:1:1:54, originalDestination = null,
>>> originalTransactionId = null, producerId =
>>> ID:ukedfrog-3905-1196171536066-1:5:1:1, destination =
>>> queue://urn:telelogic:ers:two, transactionId = null, expiration = 0,
>>> timestamp = 1196171643911, arrival = 0, correlationId = null, replyTo =
>>> queue://reply-192.168.68.151@urn:telelogic:ers:two, persistent = true,
>>> type
>>> = null, priority = 4, groupID = null, groupSequence = 0,
>>> targetConsumerId =
>>> null, compressed = false, userID = null, content = null,
>>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>>> 0,
>>> size = 0, properties = null, readOnlyProperties = true, readOnlyBody =
>>> true,
>>> droppable = false, text = messageContent}
>>> DEBUG org.apache.activemq.transport.tcp.TcpBufferedOutputStream calling
>>> flush - messageID: ID:ukedfrog-3905-1196171536066-1:5:1:1:54 count: 116
>>> OutputStream: java.net.SocketOutputStream@94124f
>>> DEBUG org.apache.activemq.transport.tcp.TcpTransport flushing bytes -
>>> messageID:ID:ukedfrog-3905-1196171536066-1:5:1:1:54 Thread:
>>> Thread[pool-1-thread-493,5,main]
>>>
>>> (The messageID is propagated to the activemq classes via a ThreadLocal)
>>>
>>> My understanding is that after TcpBufferedOutputStream.flush() the TCP
>>> packet is delegated to the underlying operating system and the message
>>> has
>>> effectively been "sent".  However, there is no receipt of this message
>>> in
>>> the broker, no matter how long we wait.  Is this message "lost" ? or is
>>> my
>>> client implementation faulty.
>>>
>>> I am using one Connection per JVM and used by each thread (with its own
>>> MessageProducer & Session). After reading
>>> http://gnodet.blogspot.com/2006/12/over-past-weeks-i-spend-some-times-load.html
>>> this  , I tried using a separate Connection per thread, but the problem
>>> persisted.  This write-up concerns me  (that the broker processes all
>>> messages on a given Connection sequentially)  as this would force the
>>> client
>>> implementation to have multiple Connections which does not seem to be
>>> mandated by the JMS spec.
>>>
>>>
>>> My ConnectionFactory is configured to use synchronous sending via
>>> "connectionFactory.setUseAsyncSend(false);"
>>>
>>> Could anybody explain why these message appear to be "lost"? It seems,
>>> from
>>> client perspective, that the message has been delivered, but it never
>>> is.
>>>
>>> Many thanks
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Messages-appear-to-get-lost-tf4881925s2354.html#a13971307
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>> 
>> 
>> -- 
>> James
>> -------
>> http://macstrac.blogspot.com/
>> 
>> Open Source Integration
>> http://open.iona.com
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Messages-appear-to-get-lost-tf4881925s2354.html#a14076065
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Re: Messages appear to get lost

Posted by ajls77 <aj...@gmail.com>.
Thanks for the response.  I have bumped by my usageManager's resource
allocation to 512Mb and the problem still persists.  Funnily enough, it
seems easier to replicate the problem with 10 threads producing every second
than 250 per second.  So it cannot be an out and out concurrency-induced.  I
am still quite suspicious that my thread model (ThreadPoolExecutor - and
some LockSupport stuff) may be causing the problem.

To verify this, could you help clarify these assumptions ?

When TcpBufferedOutputStream.flush() is called, and useAsyncSend(false), the
message, as far as the JVM is concerned, has "left the building".

and

If not, what part of ActiveMQ could affect the sending after this point.


I am convinced that TcpBufferedOutputStream.flush() is executing correctly
for a given message ID.   

Many thanks

P.S. I cannot, as of now use 5.x, but the message cursors will give a nice
amount of control over dispatching.  


James.Strachan wrote:
> 
> Maybe you're hitting the flow control causing some producers to be
> blocked. See this FAQ entry...
> http://activemq.apache.org/my-producer-blocks.html
> 
> You might wanna try using 5.x which has much better support for
> dealing with flow control in different ways.
> 
> 
> On 27/11/2007, ajls77 <aj...@gmail.com> wrote:
>>
>> I am sending messages to an ActiveMQ broker from an Java5 ExecutorService
>> thread pool.  Concurrency requirements at the moment require at least 60
>> MessageProducers - these, along with a Session, live in a thread-safe
>> FIFO
>> pipe.
>>
>> When concurrency hots up, it appears that some messages are being lost -
>> from the client's perspective it appears to have been delivered, but the
>> debugging output on the broker does not provide a receipt of the message
>> (the message delivery is set to PERSISTENT just to get the debugging as
>> there is no debugging information under NON_PERSISTENT).
>>
>> To prove that the client thinks the message is delivered I recompiled
>> activemq-core-4.1.1.jar with extra debugging.  I traced the following
>> execution flow:
>>
>> ActiveMQMessageProducer.send()
>> -> ActiveMQSession.send()
>>  -> TcpTransport.oneway()
>>    -> TcpBufferedOutputStream.flush()
>>
>> The output of which is:
>>
>> DEBUG: org.apache.activemq.ActiveMQSession - sending message with
>> messageID:
>> ID:ukedfrog-3905-1196171536066-1:5:1:1:54 Thread:
>> Thread[pool-1-thread-493,5,main]
>> DEBUG: org.apache.activemq.ActiveMQSession - Sending message:
>> ActiveMQTextMessage {commandId = 0, responseRequired = false, messageId =
>> ID:ukedfrog-3905-1196171536066-1:5:1:1:54, originalDestination = null,
>> originalTransactionId = null, producerId =
>> ID:ukedfrog-3905-1196171536066-1:5:1:1, destination =
>> queue://urn:telelogic:ers:two, transactionId = null, expiration = 0,
>> timestamp = 1196171643911, arrival = 0, correlationId = null, replyTo =
>> queue://reply-192.168.68.151@urn:telelogic:ers:two, persistent = true,
>> type
>> = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId
>> =
>> null, compressed = false, userID = null, content = null,
>> marshalledProperties = null, dataStructure = null, redeliveryCounter = 0,
>> size = 0, properties = null, readOnlyProperties = true, readOnlyBody =
>> true,
>> droppable = false, text = messageContent}
>> DEBUG org.apache.activemq.transport.tcp.TcpBufferedOutputStream calling
>> flush - messageID: ID:ukedfrog-3905-1196171536066-1:5:1:1:54 count: 116
>> OutputStream: java.net.SocketOutputStream@94124f
>> DEBUG org.apache.activemq.transport.tcp.TcpTransport flushing bytes -
>> messageID:ID:ukedfrog-3905-1196171536066-1:5:1:1:54 Thread:
>> Thread[pool-1-thread-493,5,main]
>>
>> (The messageID is propagated to the activemq classes via a ThreadLocal)
>>
>> My understanding is that after TcpBufferedOutputStream.flush() the TCP
>> packet is delegated to the underlying operating system and the message
>> has
>> effectively been "sent".  However, there is no receipt of this message in
>> the broker, no matter how long we wait.  Is this message "lost" ? or is
>> my
>> client implementation faulty.
>>
>> I am using one Connection per JVM and used by each thread (with its own
>> MessageProducer & Session). After reading
>> http://gnodet.blogspot.com/2006/12/over-past-weeks-i-spend-some-times-load.html
>> this  , I tried using a separate Connection per thread, but the problem
>> persisted.  This write-up concerns me  (that the broker processes all
>> messages on a given Connection sequentially)  as this would force the
>> client
>> implementation to have multiple Connections which does not seem to be
>> mandated by the JMS spec.
>>
>>
>> My ConnectionFactory is configured to use synchronous sending via
>> "connectionFactory.setUseAsyncSend(false);"
>>
>> Could anybody explain why these message appear to be "lost"? It seems,
>> from
>> client perspective, that the message has been delivered, but it never is.
>>
>> Many thanks
>> --
>> View this message in context:
>> http://www.nabble.com/Messages-appear-to-get-lost-tf4881925s2354.html#a13971307
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> -- 
> James
> -------
> http://macstrac.blogspot.com/
> 
> Open Source Integration
> http://open.iona.com
> 
> 

-- 
View this message in context: http://www.nabble.com/Messages-appear-to-get-lost-tf4881925s2354.html#a14058564
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Re: Messages appear to get lost

Posted by James Strachan <ja...@gmail.com>.
Maybe you're hitting the flow control causing some producers to be
blocked. See this FAQ entry...
http://activemq.apache.org/my-producer-blocks.html

You might wanna try using 5.x which has much better support for
dealing with flow control in different ways.


On 27/11/2007, ajls77 <aj...@gmail.com> wrote:
>
> I am sending messages to an ActiveMQ broker from an Java5 ExecutorService
> thread pool.  Concurrency requirements at the moment require at least 60
> MessageProducers - these, along with a Session, live in a thread-safe FIFO
> pipe.
>
> When concurrency hots up, it appears that some messages are being lost -
> from the client's perspective it appears to have been delivered, but the
> debugging output on the broker does not provide a receipt of the message
> (the message delivery is set to PERSISTENT just to get the debugging as
> there is no debugging information under NON_PERSISTENT).
>
> To prove that the client thinks the message is delivered I recompiled
> activemq-core-4.1.1.jar with extra debugging.  I traced the following
> execution flow:
>
> ActiveMQMessageProducer.send()
> -> ActiveMQSession.send()
>  -> TcpTransport.oneway()
>    -> TcpBufferedOutputStream.flush()
>
> The output of which is:
>
> DEBUG: org.apache.activemq.ActiveMQSession - sending message with messageID:
> ID:ukedfrog-3905-1196171536066-1:5:1:1:54 Thread:
> Thread[pool-1-thread-493,5,main]
> DEBUG: org.apache.activemq.ActiveMQSession - Sending message:
> ActiveMQTextMessage {commandId = 0, responseRequired = false, messageId =
> ID:ukedfrog-3905-1196171536066-1:5:1:1:54, originalDestination = null,
> originalTransactionId = null, producerId =
> ID:ukedfrog-3905-1196171536066-1:5:1:1, destination =
> queue://urn:telelogic:ers:two, transactionId = null, expiration = 0,
> timestamp = 1196171643911, arrival = 0, correlationId = null, replyTo =
> queue://reply-192.168.68.151@urn:telelogic:ers:two, persistent = true, type
> = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId =
> null, compressed = false, userID = null, content = null,
> marshalledProperties = null, dataStructure = null, redeliveryCounter = 0,
> size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true,
> droppable = false, text = messageContent}
> DEBUG org.apache.activemq.transport.tcp.TcpBufferedOutputStream calling
> flush - messageID: ID:ukedfrog-3905-1196171536066-1:5:1:1:54 count: 116
> OutputStream: java.net.SocketOutputStream@94124f
> DEBUG org.apache.activemq.transport.tcp.TcpTransport flushing bytes -
> messageID:ID:ukedfrog-3905-1196171536066-1:5:1:1:54 Thread:
> Thread[pool-1-thread-493,5,main]
>
> (The messageID is propagated to the activemq classes via a ThreadLocal)
>
> My understanding is that after TcpBufferedOutputStream.flush() the TCP
> packet is delegated to the underlying operating system and the message has
> effectively been "sent".  However, there is no receipt of this message in
> the broker, no matter how long we wait.  Is this message "lost" ? or is my
> client implementation faulty.
>
> I am using one Connection per JVM and used by each thread (with its own
> MessageProducer & Session). After reading
> http://gnodet.blogspot.com/2006/12/over-past-weeks-i-spend-some-times-load.html
> this  , I tried using a separate Connection per thread, but the problem
> persisted.  This write-up concerns me  (that the broker processes all
> messages on a given Connection sequentially)  as this would force the client
> implementation to have multiple Connections which does not seem to be
> mandated by the JMS spec.
>
>
> My ConnectionFactory is configured to use synchronous sending via
> "connectionFactory.setUseAsyncSend(false);"
>
> Could anybody explain why these message appear to be "lost"? It seems, from
> client perspective, that the message has been delivered, but it never is.
>
> Many thanks
> --
> View this message in context: http://www.nabble.com/Messages-appear-to-get-lost-tf4881925s2354.html#a13971307
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com