You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Alistair Young <Al...@uhi.ac.uk> on 2011/09/28 12:16:19 UTC

Messages being lost from route

Hi there,

is there anything that would cause camel to ignore messages in a route? I'm getting persistent messages simply disappearing from a transacted route. I've verified the messages arrive at the activemq broker (tcpdump) and also that on one occasion, 5 out of 6 of the messages just disappeared. They didn't make it into the route as the processor didn't see them and the consumer never saw them either. They just vanished. Restarting the broker seems to ease the situation until the next disappearing messages and it gets worse and worse.

thanks,

Alistair

-- 
mov eax,1
mov ebx,0
int 80h

Re: Messages being lost from route

Posted by Claus Ibsen <cl...@gmail.com>.
Yes <transacted> should be before the <process>.

We may add a validation check in the future | Camel 3.0 for that so
you cannot mis configure it.


On Thu, Sep 29, 2011 at 10:36 AM, Alistair Young
<al...@uhi.ac.uk> wrote:
>> <transacted/> Should be after <from>
> it is after from - do you mean it should be before?
>    <route id="eDirSuccessBroadcast">
>      <from uri="activemq:topic:edirectoryprocessed"/>
>      <transacted />
>      <process ref="groupwiseProcessor" />
>      <to uri="activemq:topic:blackboard"/>
>    </route>
>
> thanks for the dead letter tips, will apply them.
>
> Alistair
>
> --------------
> mov eax,1
> mov ebx,0
> int 80
>
> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>
>> Hi
>>
>> <transacted/> Should be after <from>
>>
>>
>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>> <al...@uhi.ac.uk> wrote:
>>>> Do you use message expiry?
>>> no
>>>
>>>> timestamp plugin
>>>
>>> using that
>>>
>>> activemq 5.5.0
>>> camel 2.8.0
>>> spring 3.0.5
>>>
>>> noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.
>>>
>>>    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>      <from uri="activemq:topic:matrix"/>
>>>      <process ref="matrixProcessor" />
>>>      <transacted />
>>>      <to uri="activemq:topic:edirectory"/>
>>>    </route>
>>>
>>>        <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
>>>                <property name="brokerURL" value="vm://matrixBroker?create=false"/>
>>>        </bean>
>>>
>>>        <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
>>>                <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>>        </bean>
>>>
>>>        <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>                <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>>                <property name="transacted" value="true"/>
>>>                <property name="transactionManager" ref="jmsTransactionManager"/>
>>>        </bean>
>>>
>>>  <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>    <property name="deadLetterUri" value="jms:queue:dead"/>
>>>    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
>>>  </bean>
>>>
>>>  <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
>>>    <property name="maximumRedeliveries" value="10"/>
>>>    <property name="redeliveryDelay" value="250"/>
>>>  </bean>
>>>
>>> thanks,
>>>
>>> Alistair
>>>
>>>
>>> --------------
>>> mov eax,1
>>> mov ebx,0
>>> int 80
>>>
>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>
>>>> Hi
>>>>
>>>> Do you use message expiry?
>>>> Make sure clocks between server/clients is synced as much as possible.
>>>>
>>>> There is a timestamp plugin
>>>> http://activemq.apache.org/timestampplugin.html
>>>>
>>>> And do you use queue or topic.
>>>> What version of AMQ and Camel are you using?
>>>> And how have you configured the AMQ broker, and the Camel context?
>>>>
>>>>
>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com> wrote:
>>>>> Hi
>>>>>
>>>>> Where the logs go, if it's logged at all, still depends on your logger and
>>>>> how you configured it.
>>>>>
>>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>>>
>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>
>>>>> Taariq
>>>>>
>>>>>
>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>>>>>
>>>>>> which is the best trace method to use? trace="true", or camelTracer and
>>>>>> traceFormatter beans? and where does the log end up? I've tried them all but
>>>>>> no log appears.
>>>>>>
>>>>>> Alistair
>>>>>>
>>>>>> --
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80h
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I suggest enable tracing to see exactly what happens in your route.
>>>>>>>
>>>>>>> regards, Marco
>>>>>>>
>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>> I now have a dead letter channel which is empty after losing 9 out of 10
>>>>>> messages. I also added a logging handler which logged nothing. Verified the
>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>
>>>>>>>> Alistair
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> FuseSource
>>>> Email: cibsen@fusesource.com
>>>> Web: http://fusesource.com
>>>> Twitter: davsclaus, fusenews
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
I think camel + tomcat is a non starter. The same config works perfectly in activemq 5.5.0 standalone with camel. Used embedded in a webapp in tomcat 6, only the first message makes it through the route. The rest vanish.

Alistair

-- 
mov eax,1
mov ebx,0
int 80h




On 4 Oct 2011, at 08:43, Alistair Young wrote:

> why would adding this:
> 
> <property name="transacted" value="true"/>
> 
> to this:
> 
> org.apache.activemq.camel.component.ActiveMQComponent
> 
> cause the broker to stop working? There are no transacted routes. The faster the messages come in, the more disappear. I have a very simple config that is guaranteed to lose 99/100 messages if the broker is transacted.
> 
> Is there anything special the client has to do? Surely not though, as there are no transacted routes.
> 
> Alistair
> 
> -- 
> mov eax,1
> mov ebx,0
> int 80h
> 
> 
> 
> 
> On 2 Oct 2011, at 09:43, Alistair Young wrote:
> 
>> eventually found the problem.
>> 
>> if transactions are enabled nothing works. Only the first message gets through the route, the rest disappear. The problems start by adding transacted and transactionManager to the ActiveMQComponent as per:
>> 
>> http://camel.apache.org/transactional-client.html
>> 
>> so I'm not sure how to go about fixing it. Would you have any pointers please?
>> 
>>   <bean id="poolConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
>>     <property name="maxConnections" value="8"/>
>>     <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>   </bean>
>> 
>>   <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
>>     <property name="brokerURL" value="vm://matrixBroker?create=false" />
>>   </bean>
>> 
>>   <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
>>     <property name="connectionFactory" ref="poolConnectionFactory"/>
>>   </bean>
>> 
>>   <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
>>     <property name="connectionFactory" ref="poolConnectionFactory"/>
>>     <property name="transacted" value="true"/>
>>     <property name="transactionManager" ref="jmsTransactionManager"/>
>>   </bean>
>> 
>> thanks,
>> 
>> Alistair
>> 
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>> 
>> On 30 Sep 2011, at 13:35, Alistair Young wrote:
>> 
>>> thanks Tim, that would be very helpful of you. I was about to dive into the camel source to see why the route wouldn't accept anything but if you have time to do a small test project I would be very grateful.
>>> 
>>> I'm running camel 2.8.1, activemq 5.5.0, spring 3.0.5 under tomcat. I've attached my pom.xml and camel-config.xml. mvn clean install then deploy to tomcat.
>>> 
>>> many thanks,
>>> 
>>> Alistair
>>> 
>>> -- 
>>> mov eax,1
>>> mov ebx,0
>>> int 80h
>>> <camel-config.xml>
>>> <pom.xml>
>>> 
>>> 
>>> On 30 Sep 2011, at 13:23, Tim wrote:
>>> 
>>>> Alistair. I just tried the same with an embedded activemq instance and it
>>>> works great.
>>>> Maybe this has to do with broker settings? I can setup a tiny test project
>>>> for you if you want to try that?
>>>> 
>>>> On Fri, Sep 30, 2011 at 11:27 AM, Alistair Young
>>>> <al...@uhi.ac.uk>wrote:
>>>> 
>>>>> I can now reproduce this every time. Sending 100 messages in very quick
>>>>> succession to a camel route. JMX:
>>>>> 
>>>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>  <from uri="activemq:topic:matrix"/>
>>>>>   <to uri="activemq:topic:edirectory"/>
>>>>> </route>
>>>>> 
>>>>> EnqueueCount = 100
>>>>> DispatchCount = 100
>>>>> InFlightCount = 100
>>>>> DequeueCount = 1
>>>>> 
>>>>> only 1 message ever gets through the route. That's 99 percent message loss,
>>>>> every time. This happens on two servers with the same config.
>>>>> 
>>>>> This happens with the producer on the same machine as the broker as well as
>>>>> a remote broker.
>>>>> 
>>>>> Is there anything I should be looking at to see where the other 99 messages
>>>>> are? JMX says they're in flight but all dead letter queues are empty and
>>>>> there are no errors anywhere.
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> --
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80h
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 30 Sep 2011, at 09:12, Alistair Young wrote:
>>>>> 
>>>>>> getting somewhere.
>>>>>> 
>>>>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>  <from uri="activemq:topic:matrix"/>
>>>>>>  <transacted />
>>>>>>  <process ref="matrixProcessor" />
>>>>>>  <to uri="activemq:topic:edirectory"/>
>>>>>> </route>
>>>>>> 
>>>>>> - producer clock is 2mins ahead of broker clock, timestampplugin enabled
>>>>> on broker
>>>>>> - no delay between messages at the producer = 62 out of 100 get through
>>>>> the route, consistently
>>>>>> - 3sec delay at producer, around 88 - 97 get through. That's as good as
>>>>> it gets
>>>>>> - 5sec delay at producer, 93 made it through the route
>>>>>> - remove the route and send direct to topic in activemq, no delay, no
>>>>> message loss, consistently
>>>>>> 
>>>>>> so the messages are being lost in the route. No matter what the delay in
>>>>> sending messages, some are always lost and vanish. There are no errors,
>>>>> nothing in any dead letter queue. They simply vanish. They don't even make
>>>>> it as far as the <process ref="matrixProcessor" />.
>>>>>> 
>>>>>> sending from a ruby producer that blats them out far quicker than the
>>>>> java producer is even worse. Only 1 - 3 ever get through the route. Removing
>>>>> the route and sending to the activemq topic instead, all messages get
>>>>> through no matter how fast they come.
>>>>>> 
>>>>>> Alistair
>>>>>> 
>>>>>> --
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80h
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 29 Sep 2011, at 18:10, Alistair Young wrote:
>>>>>> 
>>>>>>> no connection pool. What was disturbing was the first message sent to
>>>>> the broker after a restart and clean out of the data dir, disappeared.
>>>>> There's a similar route on the broker that works fine. The only difference
>>>>> is the producer for the wonky route is on windows and is up to 1min ahead of
>>>>> the broker's clock. Would have thought the timestamplugin would take care of
>>>>> that though. Can see it changing the timestamp in the logs.
>>>>>>> 
>>>>>>> Alistair
>>>>>>> 
>>>>>>> --------------
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80
>>>>>>> 
>>>>>>> On 29 Sep 2011, at 17:31, Tim wrote:
>>>>>>> 
>>>>>>>> Well with only 17 you definitely aren't hitting any prefetch limits or
>>>>>>>> anything like that.
>>>>>>>> Are you using a connection pool?
>>>>>>>> 
>>>>>>>> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <
>>>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>> 
>>>>>>>>> I think this way madness lies.
>>>>>>>>> 
>>>>>>>>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
>>>>>>>>> topicB enqueueCount = 12
>>>>>>>>> 
>>>>>>>>> so 17 came in, 12 made it through, of the 5 that went missing it
>>>>> claims to
>>>>>>>>> have sent 3 to topicB but they never arrived and the last 2 just
>>>>> simply
>>>>>>>>> vanished completely.
>>>>>>>>> 
>>>>>>>>> What on earth?
>>>>>>>>> 
>>>>>>>>> Alistair
>>>>>>>>> 
>>>>>>>>> --------------
>>>>>>>>> mov eax,1
>>>>>>>>> mov ebx,0
>>>>>>>>> int 80
>>>>>>>>> 
>>>>>>>>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
>>>>>>>>> 
>>>>>>>>>> nup - cleaned out the data dir and restarted the broker. First
>>>>> message in
>>>>>>>>> vanished. Wasn't persisted. So something is fundamentally broken.
>>>>>>>>>> 
>>>>>>>>>> topicA inflightCount = dispatchCount = enqueueCount = 1
>>>>>>>>>> topicB is completely empty
>>>>>>>>>> 
>>>>>>>>>> so the message wasn't persisted, wasn't processed, wasn't routed and
>>>>> just
>>>>>>>>> vanished from the broker.
>>>>>>>>>> 
>>>>>>>>>> Alistair
>>>>>>>>>> 
>>>>>>>>>> --------------
>>>>>>>>>> mov eax,1
>>>>>>>>>> mov ebx,0
>>>>>>>>>> int 80
>>>>>>>>>> 
>>>>>>>>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
>>>>>>>>>> 
>>>>>>>>>>> route goes from topicA -> topicB, transacted.
>>>>>>>>>>> topicA inflightCount = 96 and increases on each batch of incoming
>>>>>>>>> messages
>>>>>>>>>>> topicB dispatchCount = enqueueCount
>>>>>>>>>>> 
>>>>>>>>>>> wondering if the missing messages are connected to topicA
>>>>> inflightCount.
>>>>>>>>> I noticed there are two consumers for topicB. The main consumer gets
>>>>> its
>>>>>>>>> messages fine. Wonder if the second consumer is a durable topic
>>>>> consumer and
>>>>>>>>> therefore activemq is persisting its messages but it hasn't connected
>>>>> in a
>>>>>>>>> very long time. Would that cause the topic to get too big? i.e.
>>>>> messages go
>>>>>>>>> into the topic until the limit is reached. Main consumer pulls
>>>>> messages off
>>>>>>>>> and messages are able to go onto topicB again. Before consumer pulls
>>>>> and
>>>>>>>>> after limit reached, messages can't get from topicA -> topicB, hence
>>>>> the
>>>>>>>>> topicA inflightCount not zero?
>>>>>>>>>>> 
>>>>>>>>>>> Alistair
>>>>>>>>>>> 
>>>>>>>>>>> --------------
>>>>>>>>>>> mov eax,1
>>>>>>>>>>> mov ebx,0
>>>>>>>>>>> int 80
>>>>>>>>>>> 
>>>>>>>>>>> On 29 Sep 2011, at 12:17, Tim wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Sorry you might have tried this since I haven't been following this
>>>>>>>>> thread.
>>>>>>>>>>>> But can you check your jmx console.
>>>>>>>>>>>> In particular check 2 things.. the route to see if the number of
>>>>>>>>> exchanges
>>>>>>>>>>>> match what you think and how if any exchanges failed.
>>>>>>>>>>>> Also check the JMX console on activemq for the queue or topic in
>>>>>>>>> question
>>>>>>>>>>>> and see how many were enqueued vs dispatched.
>>>>>>>>>>>> Check your deadletter queue from there too
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>>>>>>>>>>> <al...@uhi.ac.uk>wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> dunno - nothing works. Random messages are just vanishing once
>>>>> they
>>>>>>>>> reach
>>>>>>>>>>>>> the broker. No trace, no logs, no dead letter queue. Just
>>>>> vanishing.
>>>>>>>>> I've
>>>>>>>>>>>>> removed <transacted /> and <process> but it doesn't help. The
>>>>> producer
>>>>>>>>> is a
>>>>>>>>>>>>> few secs behind the broker:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> sent : 11:25:26
>>>>>>>>>>>>> arrived : 11:24:57
>>>>>>>>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
>>>>>>>>> presumably
>>>>>>>>>>>>> the timestampplugin doing this
>>>>>>>>>>>>> message vanishes
>>>>>>>>>>>>> 
>>>>>>>>>>>>> but all messages display this clock behaviour and not all vanish.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --------------
>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>> int 80
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> just saw your info about transacted being before from - will
>>>>> change
>>>>>>>>> that
>>>>>>>>>>>>> and monitor again.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> just noticed a batch of identical 5 messages, three were missing
>>>>> and
>>>>>>>>>>>>> another single message vanished. tracer logged nothing. No errors,
>>>>>>>>> dead
>>>>>>>>>>>>> letter queue empty.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> One thing that happens is another machine polls the stats topic
>>>>> in
>>>>>>>>>>>>> activemq every 2mins. Would that cause a problem? It asks for
>>>>> stats on
>>>>>>>>> the
>>>>>>>>>>>>> matrix topic, which is part of the transacted route.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>>>>>>>>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
>>>>>>>>> [null]:
>>>>>>>>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>>>>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
>>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>>>>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>>>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>>>>>>>>>>>> vm://matrixBroker#285920
>>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>>>>>>>>>>>> {commandId = 1, responseRequired = true, connectionId =
>>>>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId
>>>>> =
>>>>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName
>>>>> =
>>>>>>>>> null,
>>>>>>>>>>>>> password = *****, brokerPath = null, brokerMasterConnector =
>>>>> false,
>>>>>>>>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>>>>>> it is after from - do you mean it should be before?
>>>>>>>>>>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>>>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>>>>>>>>>>> <transacted />
>>>>>>>>>>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>>>>>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>>>>>>>>>>> </route>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>>>>>>>>>>>> <al...@uhi.ac.uk> wrote:
>>>>>>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> timestamp plugin
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> using that
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> activemq 5.5.0
>>>>>>>>>>>>>>>>>> camel 2.8.0
>>>>>>>>>>>>>>>>>> spring 3.0.5
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer
>>>>> is
>>>>>>>>>>>>> logging so hopefully I can see any errors.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> <route id="matrix"
>>>>>>>>> errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>>>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>>>>>>>>>>> <transacted />
>>>>>>>>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>>>>>>>>>>> </route>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> <bean id="jmsConnectionFactory"
>>>>>>>>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>>>>>>>>>>> depends-on="matrixBrokerID">
>>>>>>>>>>>>>>>>>>   <property name="brokerURL"
>>>>>>>>>>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> <bean id="jmsTransactionManager"
>>>>>>>>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>>>>>>>>>>>   <property name="connectionFactory"
>>>>>>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> <bean id="activemq"
>>>>>>>>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>>>>>>>>>>>   <property name="connectionFactory"
>>>>>>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>>>>>>>   <property name="transacted" value="true"/>
>>>>>>>>>>>>>>>>>>   <property name="transactionManager"
>>>>>>>>>>>>> ref="jmsTransactionManager"/>
>>>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>>>>>>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>>>>>>>>>>> <property name="redeliveryPolicy"
>>>>>>>>>>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>>>>>>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>>>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>>>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>>>>>>>>>>>> And how have you configured the AMQ broker, and the Camel
>>>>>>>>> context?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
>>>>>>>>> taariql@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on
>>>>> your
>>>>>>>>>>>>> logger and
>>>>>>>>>>>>>>>>>>>> how you configured it.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Here are links to how to enable logging[1] and camel
>>>>> logging
>>>>>>>>> FAQ[2]
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> [1]
>>>>> http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Taariq
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>>>>>>>>>>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>>>>>>>>>>>> camelTracer and
>>>>>>>>>>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
>>>>>>>>> tried
>>>>>>>>>>>>> them all but
>>>>>>>>>>>>>>>>>>>>> no log appears.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>>>>>>>> int 80h
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in
>>>>> your
>>>>>>>>>>>>> route.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>>>>>>>>>>>> I now have a dead letter channel which is empty after
>>>>> losing
>>>>>>>>> 9
>>>>>>>>>>>>> out of 10
>>>>>>>>>>>>>>>>>>>>> messages. I also added a logging handler which logged
>>>>> nothing.
>>>>>>>>>>>>> Verified the
>>>>>>>>>>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>> 
>> 
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
why would adding this:

<property name="transacted" value="true"/>

to this:

org.apache.activemq.camel.component.ActiveMQComponent

cause the broker to stop working? There are no transacted routes. The faster the messages come in, the more disappear. I have a very simple config that is guaranteed to lose 99/100 messages if the broker is transacted.

Is there anything special the client has to do? Surely not though, as there are no transacted routes.

Alistair

-- 
mov eax,1
mov ebx,0
int 80h




On 2 Oct 2011, at 09:43, Alistair Young wrote:

> eventually found the problem.
> 
> if transactions are enabled nothing works. Only the first message gets through the route, the rest disappear. The problems start by adding transacted and transactionManager to the ActiveMQComponent as per:
> 
> http://camel.apache.org/transactional-client.html
> 
> so I'm not sure how to go about fixing it. Would you have any pointers please?
> 
>    <bean id="poolConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
>      <property name="maxConnections" value="8"/>
>      <property name="connectionFactory" ref="jmsConnectionFactory"/>
>    </bean>
> 
>    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
>      <property name="brokerURL" value="vm://matrixBroker?create=false" />
>    </bean>
> 
>    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
>      <property name="connectionFactory" ref="poolConnectionFactory"/>
>    </bean>
> 
>    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
>      <property name="connectionFactory" ref="poolConnectionFactory"/>
>      <property name="transacted" value="true"/>
>      <property name="transactionManager" ref="jmsTransactionManager"/>
>    </bean>
> 
> thanks,
> 
> Alistair
> 
> --------------
> mov eax,1
> mov ebx,0
> int 80
> 
> On 30 Sep 2011, at 13:35, Alistair Young wrote:
> 
>> thanks Tim, that would be very helpful of you. I was about to dive into the camel source to see why the route wouldn't accept anything but if you have time to do a small test project I would be very grateful.
>> 
>> I'm running camel 2.8.1, activemq 5.5.0, spring 3.0.5 under tomcat. I've attached my pom.xml and camel-config.xml. mvn clean install then deploy to tomcat.
>> 
>> many thanks,
>> 
>> Alistair
>> 
>> -- 
>> mov eax,1
>> mov ebx,0
>> int 80h
>> <camel-config.xml>
>> <pom.xml>
>> 
>> 
>> On 30 Sep 2011, at 13:23, Tim wrote:
>> 
>>> Alistair. I just tried the same with an embedded activemq instance and it
>>> works great.
>>> Maybe this has to do with broker settings? I can setup a tiny test project
>>> for you if you want to try that?
>>> 
>>> On Fri, Sep 30, 2011 at 11:27 AM, Alistair Young
>>> <al...@uhi.ac.uk>wrote:
>>> 
>>>> I can now reproduce this every time. Sending 100 messages in very quick
>>>> succession to a camel route. JMX:
>>>> 
>>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>   <from uri="activemq:topic:matrix"/>
>>>>    <to uri="activemq:topic:edirectory"/>
>>>> </route>
>>>> 
>>>> EnqueueCount = 100
>>>> DispatchCount = 100
>>>> InFlightCount = 100
>>>> DequeueCount = 1
>>>> 
>>>> only 1 message ever gets through the route. That's 99 percent message loss,
>>>> every time. This happens on two servers with the same config.
>>>> 
>>>> This happens with the producer on the same machine as the broker as well as
>>>> a remote broker.
>>>> 
>>>> Is there anything I should be looking at to see where the other 99 messages
>>>> are? JMX says they're in flight but all dead letter queues are empty and
>>>> there are no errors anywhere.
>>>> 
>>>> Alistair
>>>> 
>>>> --
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80h
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 30 Sep 2011, at 09:12, Alistair Young wrote:
>>>> 
>>>>> getting somewhere.
>>>>> 
>>>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>   <from uri="activemq:topic:matrix"/>
>>>>>   <transacted />
>>>>>   <process ref="matrixProcessor" />
>>>>>   <to uri="activemq:topic:edirectory"/>
>>>>> </route>
>>>>> 
>>>>> - producer clock is 2mins ahead of broker clock, timestampplugin enabled
>>>> on broker
>>>>> - no delay between messages at the producer = 62 out of 100 get through
>>>> the route, consistently
>>>>> - 3sec delay at producer, around 88 - 97 get through. That's as good as
>>>> it gets
>>>>> - 5sec delay at producer, 93 made it through the route
>>>>> - remove the route and send direct to topic in activemq, no delay, no
>>>> message loss, consistently
>>>>> 
>>>>> so the messages are being lost in the route. No matter what the delay in
>>>> sending messages, some are always lost and vanish. There are no errors,
>>>> nothing in any dead letter queue. They simply vanish. They don't even make
>>>> it as far as the <process ref="matrixProcessor" />.
>>>>> 
>>>>> sending from a ruby producer that blats them out far quicker than the
>>>> java producer is even worse. Only 1 - 3 ever get through the route. Removing
>>>> the route and sending to the activemq topic instead, all messages get
>>>> through no matter how fast they come.
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> --
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80h
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 29 Sep 2011, at 18:10, Alistair Young wrote:
>>>>> 
>>>>>> no connection pool. What was disturbing was the first message sent to
>>>> the broker after a restart and clean out of the data dir, disappeared.
>>>> There's a similar route on the broker that works fine. The only difference
>>>> is the producer for the wonky route is on windows and is up to 1min ahead of
>>>> the broker's clock. Would have thought the timestamplugin would take care of
>>>> that though. Can see it changing the timestamp in the logs.
>>>>>> 
>>>>>> Alistair
>>>>>> 
>>>>>> --------------
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80
>>>>>> 
>>>>>> On 29 Sep 2011, at 17:31, Tim wrote:
>>>>>> 
>>>>>>> Well with only 17 you definitely aren't hitting any prefetch limits or
>>>>>>> anything like that.
>>>>>>> Are you using a connection pool?
>>>>>>> 
>>>>>>> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <
>>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>> 
>>>>>>>> I think this way madness lies.
>>>>>>>> 
>>>>>>>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
>>>>>>>> topicB enqueueCount = 12
>>>>>>>> 
>>>>>>>> so 17 came in, 12 made it through, of the 5 that went missing it
>>>> claims to
>>>>>>>> have sent 3 to topicB but they never arrived and the last 2 just
>>>> simply
>>>>>>>> vanished completely.
>>>>>>>> 
>>>>>>>> What on earth?
>>>>>>>> 
>>>>>>>> Alistair
>>>>>>>> 
>>>>>>>> --------------
>>>>>>>> mov eax,1
>>>>>>>> mov ebx,0
>>>>>>>> int 80
>>>>>>>> 
>>>>>>>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
>>>>>>>> 
>>>>>>>>> nup - cleaned out the data dir and restarted the broker. First
>>>> message in
>>>>>>>> vanished. Wasn't persisted. So something is fundamentally broken.
>>>>>>>>> 
>>>>>>>>> topicA inflightCount = dispatchCount = enqueueCount = 1
>>>>>>>>> topicB is completely empty
>>>>>>>>> 
>>>>>>>>> so the message wasn't persisted, wasn't processed, wasn't routed and
>>>> just
>>>>>>>> vanished from the broker.
>>>>>>>>> 
>>>>>>>>> Alistair
>>>>>>>>> 
>>>>>>>>> --------------
>>>>>>>>> mov eax,1
>>>>>>>>> mov ebx,0
>>>>>>>>> int 80
>>>>>>>>> 
>>>>>>>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
>>>>>>>>> 
>>>>>>>>>> route goes from topicA -> topicB, transacted.
>>>>>>>>>> topicA inflightCount = 96 and increases on each batch of incoming
>>>>>>>> messages
>>>>>>>>>> topicB dispatchCount = enqueueCount
>>>>>>>>>> 
>>>>>>>>>> wondering if the missing messages are connected to topicA
>>>> inflightCount.
>>>>>>>> I noticed there are two consumers for topicB. The main consumer gets
>>>> its
>>>>>>>> messages fine. Wonder if the second consumer is a durable topic
>>>> consumer and
>>>>>>>> therefore activemq is persisting its messages but it hasn't connected
>>>> in a
>>>>>>>> very long time. Would that cause the topic to get too big? i.e.
>>>> messages go
>>>>>>>> into the topic until the limit is reached. Main consumer pulls
>>>> messages off
>>>>>>>> and messages are able to go onto topicB again. Before consumer pulls
>>>> and
>>>>>>>> after limit reached, messages can't get from topicA -> topicB, hence
>>>> the
>>>>>>>> topicA inflightCount not zero?
>>>>>>>>>> 
>>>>>>>>>> Alistair
>>>>>>>>>> 
>>>>>>>>>> --------------
>>>>>>>>>> mov eax,1
>>>>>>>>>> mov ebx,0
>>>>>>>>>> int 80
>>>>>>>>>> 
>>>>>>>>>> On 29 Sep 2011, at 12:17, Tim wrote:
>>>>>>>>>> 
>>>>>>>>>>> Sorry you might have tried this since I haven't been following this
>>>>>>>> thread.
>>>>>>>>>>> But can you check your jmx console.
>>>>>>>>>>> In particular check 2 things.. the route to see if the number of
>>>>>>>> exchanges
>>>>>>>>>>> match what you think and how if any exchanges failed.
>>>>>>>>>>> Also check the JMX console on activemq for the queue or topic in
>>>>>>>> question
>>>>>>>>>>> and see how many were enqueued vs dispatched.
>>>>>>>>>>> Check your deadletter queue from there too
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>>>>>>>>>> <al...@uhi.ac.uk>wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> dunno - nothing works. Random messages are just vanishing once
>>>> they
>>>>>>>> reach
>>>>>>>>>>>> the broker. No trace, no logs, no dead letter queue. Just
>>>> vanishing.
>>>>>>>> I've
>>>>>>>>>>>> removed <transacted /> and <process> but it doesn't help. The
>>>> producer
>>>>>>>> is a
>>>>>>>>>>>> few secs behind the broker:
>>>>>>>>>>>> 
>>>>>>>>>>>> sent : 11:25:26
>>>>>>>>>>>> arrived : 11:24:57
>>>>>>>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
>>>>>>>> presumably
>>>>>>>>>>>> the timestampplugin doing this
>>>>>>>>>>>> message vanishes
>>>>>>>>>>>> 
>>>>>>>>>>>> but all messages display this clock behaviour and not all vanish.
>>>>>>>>>>>> 
>>>>>>>>>>>> Alistair
>>>>>>>>>>>> 
>>>>>>>>>>>> --------------
>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>> int 80
>>>>>>>>>>>> 
>>>>>>>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> just saw your info about transacted being before from - will
>>>> change
>>>>>>>> that
>>>>>>>>>>>> and monitor again.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --------------
>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>> int 80
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> just noticed a batch of identical 5 messages, three were missing
>>>> and
>>>>>>>>>>>> another single message vanished. tracer logged nothing. No errors,
>>>>>>>> dead
>>>>>>>>>>>> letter queue empty.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> One thing that happens is another machine polls the stats topic
>>>> in
>>>>>>>>>>>> activemq every 2mins. Would that cause a problem? It asks for
>>>> stats on
>>>>>>>> the
>>>>>>>>>>>> matrix topic, which is part of the transacted route.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>>>>>>>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
>>>>>>>> [null]:
>>>>>>>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>>>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>>>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>>>>>>>>>>> vm://matrixBroker#285920
>>>>>>>>>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>>>>>>>>>>> {commandId = 1, responseRequired = true, connectionId =
>>>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId
>>>> =
>>>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName
>>>> =
>>>>>>>> null,
>>>>>>>>>>>> password = *****, brokerPath = null, brokerMasterConnector =
>>>> false,
>>>>>>>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>>>>> it is after from - do you mean it should be before?
>>>>>>>>>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>>>>>>>>>> <transacted />
>>>>>>>>>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>>>>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>>>>>>>>>> </route>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>>>>>>>>>>> <al...@uhi.ac.uk> wrote:
>>>>>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> timestamp plugin
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> using that
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> activemq 5.5.0
>>>>>>>>>>>>>>>>> camel 2.8.0
>>>>>>>>>>>>>>>>> spring 3.0.5
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer
>>>> is
>>>>>>>>>>>> logging so hopefully I can see any errors.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> <route id="matrix"
>>>>>>>> errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>>>>>>>>>> <transacted />
>>>>>>>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>>>>>>>>>> </route>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> <bean id="jmsConnectionFactory"
>>>>>>>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>>>>>>>>>> depends-on="matrixBrokerID">
>>>>>>>>>>>>>>>>>    <property name="brokerURL"
>>>>>>>>>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> <bean id="jmsTransactionManager"
>>>>>>>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>>>>>>>>>>    <property name="connectionFactory"
>>>>>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> <bean id="activemq"
>>>>>>>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>>>>>>>>>>    <property name="connectionFactory"
>>>>>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>>>>>>    <property name="transacted" value="true"/>
>>>>>>>>>>>>>>>>>    <property name="transactionManager"
>>>>>>>>>>>> ref="jmsTransactionManager"/>
>>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>>>>>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>>>>>>>>>> <property name="redeliveryPolicy"
>>>>>>>>>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>>>>>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>>>>>>>>>>> And how have you configured the AMQ broker, and the Camel
>>>>>>>> context?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
>>>>>>>> taariql@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on
>>>> your
>>>>>>>>>>>> logger and
>>>>>>>>>>>>>>>>>>> how you configured it.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Here are links to how to enable logging[1] and camel
>>>> logging
>>>>>>>> FAQ[2]
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> [1]
>>>> http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Taariq
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>>>>>>>>>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>>>>>>>>>>> camelTracer and
>>>>>>>>>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
>>>>>>>> tried
>>>>>>>>>>>> them all but
>>>>>>>>>>>>>>>>>>>> no log appears.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>>>>>>> int 80h
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in
>>>> your
>>>>>>>>>>>> route.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>>>>>>>>>>> I now have a dead letter channel which is empty after
>>>> losing
>>>>>>>> 9
>>>>>>>>>>>> out of 10
>>>>>>>>>>>>>>>>>>>> messages. I also added a logging handler which logged
>>>> nothing.
>>>>>>>>>>>> Verified the
>>>>>>>>>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>> 
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
eventually found the problem.

if transactions are enabled nothing works. Only the first message gets through the route, the rest disappear. The problems start by adding transacted and transactionManager to the ActiveMQComponent as per:

http://camel.apache.org/transactional-client.html

so I'm not sure how to go about fixing it. Would you have any pointers please?

    <bean id="poolConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
      <property name="maxConnections" value="8"/>
      <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
      <property name="brokerURL" value="vm://matrixBroker?create=false" />
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
      <property name="connectionFactory" ref="poolConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
      <property name="connectionFactory" ref="poolConnectionFactory"/>
      <property name="transacted" value="true"/>
      <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

thanks,

Alistair

--------------
mov eax,1
mov ebx,0
int 80

On 30 Sep 2011, at 13:35, Alistair Young wrote:

> thanks Tim, that would be very helpful of you. I was about to dive into the camel source to see why the route wouldn't accept anything but if you have time to do a small test project I would be very grateful.
> 
> I'm running camel 2.8.1, activemq 5.5.0, spring 3.0.5 under tomcat. I've attached my pom.xml and camel-config.xml. mvn clean install then deploy to tomcat.
> 
> many thanks,
> 
> Alistair
> 
> -- 
> mov eax,1
> mov ebx,0
> int 80h
> <camel-config.xml>
> <pom.xml>
> 
> 
> On 30 Sep 2011, at 13:23, Tim wrote:
> 
>> Alistair. I just tried the same with an embedded activemq instance and it
>> works great.
>> Maybe this has to do with broker settings? I can setup a tiny test project
>> for you if you want to try that?
>> 
>> On Fri, Sep 30, 2011 at 11:27 AM, Alistair Young
>> <al...@uhi.ac.uk>wrote:
>> 
>>> I can now reproduce this every time. Sending 100 messages in very quick
>>> succession to a camel route. JMX:
>>> 
>>>  <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>    <from uri="activemq:topic:matrix"/>
>>>     <to uri="activemq:topic:edirectory"/>
>>>  </route>
>>> 
>>> EnqueueCount = 100
>>> DispatchCount = 100
>>> InFlightCount = 100
>>> DequeueCount = 1
>>> 
>>> only 1 message ever gets through the route. That's 99 percent message loss,
>>> every time. This happens on two servers with the same config.
>>> 
>>> This happens with the producer on the same machine as the broker as well as
>>> a remote broker.
>>> 
>>> Is there anything I should be looking at to see where the other 99 messages
>>> are? JMX says they're in flight but all dead letter queues are empty and
>>> there are no errors anywhere.
>>> 
>>> Alistair
>>> 
>>> --
>>> mov eax,1
>>> mov ebx,0
>>> int 80h
>>> 
>>> 
>>> 
>>> 
>>> On 30 Sep 2011, at 09:12, Alistair Young wrote:
>>> 
>>>> getting somewhere.
>>>> 
>>>>  <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>    <from uri="activemq:topic:matrix"/>
>>>>    <transacted />
>>>>    <process ref="matrixProcessor" />
>>>>    <to uri="activemq:topic:edirectory"/>
>>>>  </route>
>>>> 
>>>> - producer clock is 2mins ahead of broker clock, timestampplugin enabled
>>> on broker
>>>> - no delay between messages at the producer = 62 out of 100 get through
>>> the route, consistently
>>>> - 3sec delay at producer, around 88 - 97 get through. That's as good as
>>> it gets
>>>> - 5sec delay at producer, 93 made it through the route
>>>> - remove the route and send direct to topic in activemq, no delay, no
>>> message loss, consistently
>>>> 
>>>> so the messages are being lost in the route. No matter what the delay in
>>> sending messages, some are always lost and vanish. There are no errors,
>>> nothing in any dead letter queue. They simply vanish. They don't even make
>>> it as far as the <process ref="matrixProcessor" />.
>>>> 
>>>> sending from a ruby producer that blats them out far quicker than the
>>> java producer is even worse. Only 1 - 3 ever get through the route. Removing
>>> the route and sending to the activemq topic instead, all messages get
>>> through no matter how fast they come.
>>>> 
>>>> Alistair
>>>> 
>>>> --
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80h
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 29 Sep 2011, at 18:10, Alistair Young wrote:
>>>> 
>>>>> no connection pool. What was disturbing was the first message sent to
>>> the broker after a restart and clean out of the data dir, disappeared.
>>> There's a similar route on the broker that works fine. The only difference
>>> is the producer for the wonky route is on windows and is up to 1min ahead of
>>> the broker's clock. Would have thought the timestamplugin would take care of
>>> that though. Can see it changing the timestamp in the logs.
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> --------------
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80
>>>>> 
>>>>> On 29 Sep 2011, at 17:31, Tim wrote:
>>>>> 
>>>>>> Well with only 17 you definitely aren't hitting any prefetch limits or
>>>>>> anything like that.
>>>>>> Are you using a connection pool?
>>>>>> 
>>>>>> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <
>>> alistair.young@uhi.ac.uk>wrote:
>>>>>> 
>>>>>>> I think this way madness lies.
>>>>>>> 
>>>>>>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
>>>>>>> topicB enqueueCount = 12
>>>>>>> 
>>>>>>> so 17 came in, 12 made it through, of the 5 that went missing it
>>> claims to
>>>>>>> have sent 3 to topicB but they never arrived and the last 2 just
>>> simply
>>>>>>> vanished completely.
>>>>>>> 
>>>>>>> What on earth?
>>>>>>> 
>>>>>>> Alistair
>>>>>>> 
>>>>>>> --------------
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80
>>>>>>> 
>>>>>>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
>>>>>>> 
>>>>>>>> nup - cleaned out the data dir and restarted the broker. First
>>> message in
>>>>>>> vanished. Wasn't persisted. So something is fundamentally broken.
>>>>>>>> 
>>>>>>>> topicA inflightCount = dispatchCount = enqueueCount = 1
>>>>>>>> topicB is completely empty
>>>>>>>> 
>>>>>>>> so the message wasn't persisted, wasn't processed, wasn't routed and
>>> just
>>>>>>> vanished from the broker.
>>>>>>>> 
>>>>>>>> Alistair
>>>>>>>> 
>>>>>>>> --------------
>>>>>>>> mov eax,1
>>>>>>>> mov ebx,0
>>>>>>>> int 80
>>>>>>>> 
>>>>>>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
>>>>>>>> 
>>>>>>>>> route goes from topicA -> topicB, transacted.
>>>>>>>>> topicA inflightCount = 96 and increases on each batch of incoming
>>>>>>> messages
>>>>>>>>> topicB dispatchCount = enqueueCount
>>>>>>>>> 
>>>>>>>>> wondering if the missing messages are connected to topicA
>>> inflightCount.
>>>>>>> I noticed there are two consumers for topicB. The main consumer gets
>>> its
>>>>>>> messages fine. Wonder if the second consumer is a durable topic
>>> consumer and
>>>>>>> therefore activemq is persisting its messages but it hasn't connected
>>> in a
>>>>>>> very long time. Would that cause the topic to get too big? i.e.
>>> messages go
>>>>>>> into the topic until the limit is reached. Main consumer pulls
>>> messages off
>>>>>>> and messages are able to go onto topicB again. Before consumer pulls
>>> and
>>>>>>> after limit reached, messages can't get from topicA -> topicB, hence
>>> the
>>>>>>> topicA inflightCount not zero?
>>>>>>>>> 
>>>>>>>>> Alistair
>>>>>>>>> 
>>>>>>>>> --------------
>>>>>>>>> mov eax,1
>>>>>>>>> mov ebx,0
>>>>>>>>> int 80
>>>>>>>>> 
>>>>>>>>> On 29 Sep 2011, at 12:17, Tim wrote:
>>>>>>>>> 
>>>>>>>>>> Sorry you might have tried this since I haven't been following this
>>>>>>> thread.
>>>>>>>>>> But can you check your jmx console.
>>>>>>>>>> In particular check 2 things.. the route to see if the number of
>>>>>>> exchanges
>>>>>>>>>> match what you think and how if any exchanges failed.
>>>>>>>>>> Also check the JMX console on activemq for the queue or topic in
>>>>>>> question
>>>>>>>>>> and see how many were enqueued vs dispatched.
>>>>>>>>>> Check your deadletter queue from there too
>>>>>>>>>> 
>>>>>>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>>>>>>>>> <al...@uhi.ac.uk>wrote:
>>>>>>>>>> 
>>>>>>>>>>> dunno - nothing works. Random messages are just vanishing once
>>> they
>>>>>>> reach
>>>>>>>>>>> the broker. No trace, no logs, no dead letter queue. Just
>>> vanishing.
>>>>>>> I've
>>>>>>>>>>> removed <transacted /> and <process> but it doesn't help. The
>>> producer
>>>>>>> is a
>>>>>>>>>>> few secs behind the broker:
>>>>>>>>>>> 
>>>>>>>>>>> sent : 11:25:26
>>>>>>>>>>> arrived : 11:24:57
>>>>>>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
>>>>>>> presumably
>>>>>>>>>>> the timestampplugin doing this
>>>>>>>>>>> message vanishes
>>>>>>>>>>> 
>>>>>>>>>>> but all messages display this clock behaviour and not all vanish.
>>>>>>>>>>> 
>>>>>>>>>>> Alistair
>>>>>>>>>>> 
>>>>>>>>>>> --------------
>>>>>>>>>>> mov eax,1
>>>>>>>>>>> mov ebx,0
>>>>>>>>>>> int 80
>>>>>>>>>>> 
>>>>>>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> just saw your info about transacted being before from - will
>>> change
>>>>>>> that
>>>>>>>>>>> and monitor again.
>>>>>>>>>>>> 
>>>>>>>>>>>> thanks,
>>>>>>>>>>>> 
>>>>>>>>>>>> Alistair
>>>>>>>>>>>> 
>>>>>>>>>>>> --------------
>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>> int 80
>>>>>>>>>>>> 
>>>>>>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> just noticed a batch of identical 5 messages, three were missing
>>> and
>>>>>>>>>>> another single message vanished. tracer logged nothing. No errors,
>>>>>>> dead
>>>>>>>>>>> letter queue empty.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> One thing that happens is another machine polls the stats topic
>>> in
>>>>>>>>>>> activemq every 2mins. Would that cause a problem? It asks for
>>> stats on
>>>>>>> the
>>>>>>>>>>> matrix topic, which is part of the transacted route.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>>>>>>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
>>>>>>> [null]:
>>>>>>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>>>>>>>>>> vm://matrixBroker#285920
>>>>>>>>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>>>>>>>>>> {commandId = 1, responseRequired = true, connectionId =
>>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId
>>> =
>>>>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName
>>> =
>>>>>>> null,
>>>>>>>>>>> password = *****, brokerPath = null, brokerMasterConnector =
>>> false,
>>>>>>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --------------
>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>> int 80
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>>>> it is after from - do you mean it should be before?
>>>>>>>>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>>>>>>>>> <transacted />
>>>>>>>>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>>>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>>>>>>>>> </route>
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>>>>>>>>>> <al...@uhi.ac.uk> wrote:
>>>>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> timestamp plugin
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> using that
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> activemq 5.5.0
>>>>>>>>>>>>>>>> camel 2.8.0
>>>>>>>>>>>>>>>> spring 3.0.5
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer
>>> is
>>>>>>>>>>> logging so hopefully I can see any errors.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> <route id="matrix"
>>>>>>> errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>>>>>>>>> <transacted />
>>>>>>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>>>>>>>>> </route>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> <bean id="jmsConnectionFactory"
>>>>>>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>>>>>>>>> depends-on="matrixBrokerID">
>>>>>>>>>>>>>>>>     <property name="brokerURL"
>>>>>>>>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> <bean id="jmsTransactionManager"
>>>>>>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>>>>>>>>>     <property name="connectionFactory"
>>>>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> <bean id="activemq"
>>>>>>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>>>>>>>>>     <property name="connectionFactory"
>>>>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>>>>>     <property name="transacted" value="true"/>
>>>>>>>>>>>>>>>>     <property name="transactionManager"
>>>>>>>>>>> ref="jmsTransactionManager"/>
>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>>>>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>>>>>>>>> <property name="redeliveryPolicy"
>>>>>>>>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>>>>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> --------------
>>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>>> int 80
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>>>>>>>>>> possible.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>>>>>>>>>> And how have you configured the AMQ broker, and the Camel
>>>>>>> context?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
>>>>>>> taariql@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on
>>> your
>>>>>>>>>>> logger and
>>>>>>>>>>>>>>>>>> how you configured it.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Here are links to how to enable logging[1] and camel
>>> logging
>>>>>>> FAQ[2]
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> [1]
>>> http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Taariq
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>>>>>>>>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>>>>>>>>>> camelTracer and
>>>>>>>>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
>>>>>>> tried
>>>>>>>>>>> them all but
>>>>>>>>>>>>>>>>>>> no log appears.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>>>>>> int 80h
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in
>>> your
>>>>>>>>>>> route.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>>>>>>>>>> I now have a dead letter channel which is empty after
>>> losing
>>>>>>> 9
>>>>>>>>>>> out of 10
>>>>>>>>>>>>>>>>>>> messages. I also added a logging handler which logged
>>> nothing.
>>>>>>>>>>> Verified the
>>>>>>>>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
thanks Tim, that would be very helpful of you. I was about to dive into the camel source to see why the route wouldn't accept anything but if you have time to do a small test project I would be very grateful.

I'm running camel 2.8.1, activemq 5.5.0, spring 3.0.5 under tomcat. I've attached my pom.xml and camel-config.xml. mvn clean install then deploy to tomcat.

many thanks,

Alistair

-- 
mov eax,1
mov ebx,0
int 80h

Re: Messages being lost from route

Posted by Tim <ch...@gmail.com>.
Alistair. I just tried the same with an embedded activemq instance and it
works great.
Maybe this has to do with broker settings? I can setup a tiny test project
for you if you want to try that?

On Fri, Sep 30, 2011 at 11:27 AM, Alistair Young
<al...@uhi.ac.uk>wrote:

> I can now reproduce this every time. Sending 100 messages in very quick
> succession to a camel route. JMX:
>
>    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>      <from uri="activemq:topic:matrix"/>
>       <to uri="activemq:topic:edirectory"/>
>    </route>
>
> EnqueueCount = 100
> DispatchCount = 100
> InFlightCount = 100
> DequeueCount = 1
>
> only 1 message ever gets through the route. That's 99 percent message loss,
> every time. This happens on two servers with the same config.
>
> This happens with the producer on the same machine as the broker as well as
> a remote broker.
>
> Is there anything I should be looking at to see where the other 99 messages
> are? JMX says they're in flight but all dead letter queues are empty and
> there are no errors anywhere.
>
> Alistair
>
> --
> mov eax,1
> mov ebx,0
> int 80h
>
>
>
>
> On 30 Sep 2011, at 09:12, Alistair Young wrote:
>
> > getting somewhere.
> >
> >    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
> >      <from uri="activemq:topic:matrix"/>
> >      <transacted />
> >      <process ref="matrixProcessor" />
> >      <to uri="activemq:topic:edirectory"/>
> >    </route>
> >
> > - producer clock is 2mins ahead of broker clock, timestampplugin enabled
> on broker
> > - no delay between messages at the producer = 62 out of 100 get through
> the route, consistently
> > - 3sec delay at producer, around 88 - 97 get through. That's as good as
> it gets
> > - 5sec delay at producer, 93 made it through the route
> > - remove the route and send direct to topic in activemq, no delay, no
> message loss, consistently
> >
> > so the messages are being lost in the route. No matter what the delay in
> sending messages, some are always lost and vanish. There are no errors,
> nothing in any dead letter queue. They simply vanish. They don't even make
> it as far as the <process ref="matrixProcessor" />.
> >
> > sending from a ruby producer that blats them out far quicker than the
> java producer is even worse. Only 1 - 3 ever get through the route. Removing
> the route and sending to the activemq topic instead, all messages get
> through no matter how fast they come.
> >
> > Alistair
> >
> > --
> > mov eax,1
> > mov ebx,0
> > int 80h
> >
> >
> >
> >
> > On 29 Sep 2011, at 18:10, Alistair Young wrote:
> >
> >> no connection pool. What was disturbing was the first message sent to
> the broker after a restart and clean out of the data dir, disappeared.
> There's a similar route on the broker that works fine. The only difference
> is the producer for the wonky route is on windows and is up to 1min ahead of
> the broker's clock. Would have thought the timestamplugin would take care of
> that though. Can see it changing the timestamp in the logs.
> >>
> >> Alistair
> >>
> >> --------------
> >> mov eax,1
> >> mov ebx,0
> >> int 80
> >>
> >> On 29 Sep 2011, at 17:31, Tim wrote:
> >>
> >>> Well with only 17 you definitely aren't hitting any prefetch limits or
> >>> anything like that.
> >>> Are you using a connection pool?
> >>>
> >>> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <
> alistair.young@uhi.ac.uk>wrote:
> >>>
> >>>> I think this way madness lies.
> >>>>
> >>>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
> >>>> topicB enqueueCount = 12
> >>>>
> >>>> so 17 came in, 12 made it through, of the 5 that went missing it
> claims to
> >>>> have sent 3 to topicB but they never arrived and the last 2 just
> simply
> >>>> vanished completely.
> >>>>
> >>>> What on earth?
> >>>>
> >>>> Alistair
> >>>>
> >>>> --------------
> >>>> mov eax,1
> >>>> mov ebx,0
> >>>> int 80
> >>>>
> >>>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
> >>>>
> >>>>> nup - cleaned out the data dir and restarted the broker. First
> message in
> >>>> vanished. Wasn't persisted. So something is fundamentally broken.
> >>>>>
> >>>>> topicA inflightCount = dispatchCount = enqueueCount = 1
> >>>>> topicB is completely empty
> >>>>>
> >>>>> so the message wasn't persisted, wasn't processed, wasn't routed and
> just
> >>>> vanished from the broker.
> >>>>>
> >>>>> Alistair
> >>>>>
> >>>>> --------------
> >>>>> mov eax,1
> >>>>> mov ebx,0
> >>>>> int 80
> >>>>>
> >>>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
> >>>>>
> >>>>>> route goes from topicA -> topicB, transacted.
> >>>>>> topicA inflightCount = 96 and increases on each batch of incoming
> >>>> messages
> >>>>>> topicB dispatchCount = enqueueCount
> >>>>>>
> >>>>>> wondering if the missing messages are connected to topicA
> inflightCount.
> >>>> I noticed there are two consumers for topicB. The main consumer gets
> its
> >>>> messages fine. Wonder if the second consumer is a durable topic
> consumer and
> >>>> therefore activemq is persisting its messages but it hasn't connected
> in a
> >>>> very long time. Would that cause the topic to get too big? i.e.
> messages go
> >>>> into the topic until the limit is reached. Main consumer pulls
> messages off
> >>>> and messages are able to go onto topicB again. Before consumer pulls
> and
> >>>> after limit reached, messages can't get from topicA -> topicB, hence
> the
> >>>> topicA inflightCount not zero?
> >>>>>>
> >>>>>> Alistair
> >>>>>>
> >>>>>> --------------
> >>>>>> mov eax,1
> >>>>>> mov ebx,0
> >>>>>> int 80
> >>>>>>
> >>>>>> On 29 Sep 2011, at 12:17, Tim wrote:
> >>>>>>
> >>>>>>> Sorry you might have tried this since I haven't been following this
> >>>> thread.
> >>>>>>> But can you check your jmx console.
> >>>>>>> In particular check 2 things.. the route to see if the number of
> >>>> exchanges
> >>>>>>> match what you think and how if any exchanges failed.
> >>>>>>> Also check the JMX console on activemq for the queue or topic in
> >>>> question
> >>>>>>> and see how many were enqueued vs dispatched.
> >>>>>>> Check your deadletter queue from there too
> >>>>>>>
> >>>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
> >>>>>>> <al...@uhi.ac.uk>wrote:
> >>>>>>>
> >>>>>>>> dunno - nothing works. Random messages are just vanishing once
> they
> >>>> reach
> >>>>>>>> the broker. No trace, no logs, no dead letter queue. Just
> vanishing.
> >>>> I've
> >>>>>>>> removed <transacted /> and <process> but it doesn't help. The
> producer
> >>>> is a
> >>>>>>>> few secs behind the broker:
> >>>>>>>>
> >>>>>>>> sent : 11:25:26
> >>>>>>>> arrived : 11:24:57
> >>>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
> >>>> presumably
> >>>>>>>> the timestampplugin doing this
> >>>>>>>> message vanishes
> >>>>>>>>
> >>>>>>>> but all messages display this clock behaviour and not all vanish.
> >>>>>>>>
> >>>>>>>> Alistair
> >>>>>>>>
> >>>>>>>> --------------
> >>>>>>>> mov eax,1
> >>>>>>>> mov ebx,0
> >>>>>>>> int 80
> >>>>>>>>
> >>>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
> >>>>>>>>
> >>>>>>>>> just saw your info about transacted being before from - will
> change
> >>>> that
> >>>>>>>> and monitor again.
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>>
> >>>>>>>>> Alistair
> >>>>>>>>>
> >>>>>>>>> --------------
> >>>>>>>>> mov eax,1
> >>>>>>>>> mov ebx,0
> >>>>>>>>> int 80
> >>>>>>>>>
> >>>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
> >>>>>>>>>
> >>>>>>>>>> just noticed a batch of identical 5 messages, three were missing
> and
> >>>>>>>> another single message vanished. tracer logged nothing. No errors,
> >>>> dead
> >>>>>>>> letter queue empty.
> >>>>>>>>>>
> >>>>>>>>>> One thing that happens is another machine polls the stats topic
> in
> >>>>>>>> activemq every 2mins. Would that cause a problem? It asks for
> stats on
> >>>> the
> >>>>>>>> matrix topic, which is part of the transacted route.
> >>>>>>>>>>
> >>>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
> >>>>>>>> Topic:ActiveMQ.Advisory.Connection
> >>>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
> >>>> [null]:
> >>>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
> >>>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
> >>>>>>>> vm://matrixBroker#285916
> >>>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
> >>>> vm://matrixBroker#285916
> >>>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
> >>>>>>>> vm://matrixBroker#285916
> >>>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
> >>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
> >>>>>>>> vm://matrixBroker#285920
> >>>>>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
> >>>>>>>> {commandId = 1, responseRequired = true, connectionId =
> >>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId
> =
> >>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName
> =
> >>>> null,
> >>>>>>>> password = *****, brokerPath = null, brokerMasterConnector =
> false,
> >>>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
> >>>>>>>>>>
> >>>>>>>>>> Alistair
> >>>>>>>>>>
> >>>>>>>>>> --------------
> >>>>>>>>>> mov eax,1
> >>>>>>>>>> mov ebx,0
> >>>>>>>>>> int 80
> >>>>>>>>>>
> >>>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
> >>>>>>>>>>
> >>>>>>>>>>>> <transacted/> Should be after <from>
> >>>>>>>>>>> it is after from - do you mean it should be before?
> >>>>>>>>>>> <route id="eDirSuccessBroadcast">
> >>>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
> >>>>>>>>>>> <transacted />
> >>>>>>>>>>> <process ref="groupwiseProcessor" />
> >>>>>>>>>>> <to uri="activemq:topic:blackboard"/>
> >>>>>>>>>>> </route>
> >>>>>>>>>>>
> >>>>>>>>>>> thanks for the dead letter tips, will apply them.
> >>>>>>>>>>>
> >>>>>>>>>>> Alistair
> >>>>>>>>>>>
> >>>>>>>>>>> --------------
> >>>>>>>>>>> mov eax,1
> >>>>>>>>>>> mov ebx,0
> >>>>>>>>>>> int 80
> >>>>>>>>>>>
> >>>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi
> >>>>>>>>>>>>
> >>>>>>>>>>>> <transacted/> Should be after <from>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
> >>>>>>>>>>>> <al...@uhi.ac.uk> wrote:
> >>>>>>>>>>>>>> Do you use message expiry?
> >>>>>>>>>>>>> no
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> timestamp plugin
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> using that
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> activemq 5.5.0
> >>>>>>>>>>>>> camel 2.8.0
> >>>>>>>>>>>>> spring 3.0.5
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer
> is
> >>>>>>>> logging so hopefully I can see any errors.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <route id="matrix"
> >>>> errorHandlerRef="matrixDeadLetterErrorHandler">
> >>>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
> >>>>>>>>>>>>> <process ref="matrixProcessor" />
> >>>>>>>>>>>>> <transacted />
> >>>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
> >>>>>>>>>>>>> </route>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <bean id="jmsConnectionFactory"
> >>>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
> >>>>>>>> depends-on="matrixBrokerID">
> >>>>>>>>>>>>>       <property name="brokerURL"
> >>>>>>>> value="vm://matrixBroker?create=false"/>
> >>>>>>>>>>>>> </bean>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <bean id="jmsTransactionManager"
> >>>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
> >>>>>>>>>>>>>       <property name="connectionFactory"
> >>>>>>>> ref="jmsConnectionFactory"/>
> >>>>>>>>>>>>> </bean>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <bean id="activemq"
> >>>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
> >>>>>>>>>>>>>       <property name="connectionFactory"
> >>>>>>>> ref="jmsConnectionFactory"/>
> >>>>>>>>>>>>>       <property name="transacted" value="true"/>
> >>>>>>>>>>>>>       <property name="transactionManager"
> >>>>>>>> ref="jmsTransactionManager"/>
> >>>>>>>>>>>>> </bean>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
> >>>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
> >>>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
> >>>>>>>>>>>>> <property name="redeliveryPolicy"
> >>>>>>>> ref="matrixRedeliveryPolicyConfig"/>
> >>>>>>>>>>>>> </bean>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
> >>>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
> >>>>>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
> >>>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
> >>>>>>>>>>>>> </bean>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> thanks,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Alistair
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --------------
> >>>>>>>>>>>>> mov eax,1
> >>>>>>>>>>>>> mov ebx,0
> >>>>>>>>>>>>> int 80
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Do you use message expiry?
> >>>>>>>>>>>>>> Make sure clocks between server/clients is synced as much as
> >>>>>>>> possible.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> There is a timestamp plugin
> >>>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> And do you use queue or topic.
> >>>>>>>>>>>>>> What version of AMQ and Camel are you using?
> >>>>>>>>>>>>>> And how have you configured the AMQ broker, and the Camel
> >>>> context?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
> >>>> taariql@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>>> Hi
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on
> your
> >>>>>>>> logger and
> >>>>>>>>>>>>>>> how you configured it.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Here are links to how to enable logging[1] and camel
> logging
> >>>> FAQ[2]
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [1]
> http://camel.apache.org/how-do-i-enable-debug-logging.html
> >>>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Taariq
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
> >>>>>>>> alistair.young@uhi.ac.uk>wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> which is the best trace method to use? trace="true", or
> >>>>>>>> camelTracer and
> >>>>>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
> >>>> tried
> >>>>>>>> them all but
> >>>>>>>>>>>>>>>> no log appears.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Alistair
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> mov eax,1
> >>>>>>>>>>>>>>>> mov ebx,0
> >>>>>>>>>>>>>>>> int 80h
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in
> your
> >>>>>>>> route.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> regards, Marco
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
> >>>>>>>>>>>>>>>>>> I now have a dead letter channel which is empty after
> losing
> >>>> 9
> >>>>>>>> out of 10
> >>>>>>>>>>>>>>>> messages. I also added a logging handler which logged
> nothing.
> >>>>>>>> Verified the
> >>>>>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Alistair
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Claus Ibsen
> >>>>>>>>>>>>>> -----------------
> >>>>>>>>>>>>>> FuseSource
> >>>>>>>>>>>>>> Email: cibsen@fusesource.com
> >>>>>>>>>>>>>> Web: http://fusesource.com
> >>>>>>>>>>>>>> Twitter: davsclaus, fusenews
> >>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
> >>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Claus Ibsen
> >>>>>>>>>>>> -----------------
> >>>>>>>>>>>> FuseSource
> >>>>>>>>>>>> Email: cibsen@fusesource.com
> >>>>>>>>>>>> Web: http://fusesource.com
> >>>>>>>>>>>> Twitter: davsclaus, fusenews
> >>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
> >>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>
> >
>
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
I can now reproduce this every time. Sending 100 messages in very quick succession to a camel route. JMX:

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
      <from uri="activemq:topic:matrix"/>
      <to uri="activemq:topic:edirectory"/>
    </route>

EnqueueCount = 100
DispatchCount = 100
InFlightCount = 100
DequeueCount = 1

only 1 message ever gets through the route. That's 99 percent message loss, every time. This happens on two servers with the same config.

This happens with the producer on the same machine as the broker as well as a remote broker.

Is there anything I should be looking at to see where the other 99 messages are? JMX says they're in flight but all dead letter queues are empty and there are no errors anywhere.

Alistair

-- 
mov eax,1
mov ebx,0
int 80h




On 30 Sep 2011, at 09:12, Alistair Young wrote:

> getting somewhere.
> 
>    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>      <from uri="activemq:topic:matrix"/>
>      <transacted />
>      <process ref="matrixProcessor" />
>      <to uri="activemq:topic:edirectory"/>
>    </route>
> 
> - producer clock is 2mins ahead of broker clock, timestampplugin enabled on broker
> - no delay between messages at the producer = 62 out of 100 get through the route, consistently
> - 3sec delay at producer, around 88 - 97 get through. That's as good as it gets
> - 5sec delay at producer, 93 made it through the route
> - remove the route and send direct to topic in activemq, no delay, no message loss, consistently
> 
> so the messages are being lost in the route. No matter what the delay in sending messages, some are always lost and vanish. There are no errors, nothing in any dead letter queue. They simply vanish. They don't even make it as far as the <process ref="matrixProcessor" />.
> 
> sending from a ruby producer that blats them out far quicker than the java producer is even worse. Only 1 - 3 ever get through the route. Removing the route and sending to the activemq topic instead, all messages get through no matter how fast they come.
> 
> Alistair
> 
> -- 
> mov eax,1
> mov ebx,0
> int 80h
> 
> 
> 
> 
> On 29 Sep 2011, at 18:10, Alistair Young wrote:
> 
>> no connection pool. What was disturbing was the first message sent to the broker after a restart and clean out of the data dir, disappeared. There's a similar route on the broker that works fine. The only difference is the producer for the wonky route is on windows and is up to 1min ahead of the broker's clock. Would have thought the timestamplugin would take care of that though. Can see it changing the timestamp in the logs.
>> 
>> Alistair
>> 
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>> 
>> On 29 Sep 2011, at 17:31, Tim wrote:
>> 
>>> Well with only 17 you definitely aren't hitting any prefetch limits or
>>> anything like that.
>>> Are you using a connection pool?
>>> 
>>> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>>> 
>>>> I think this way madness lies.
>>>> 
>>>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
>>>> topicB enqueueCount = 12
>>>> 
>>>> so 17 came in, 12 made it through, of the 5 that went missing it claims to
>>>> have sent 3 to topicB but they never arrived and the last 2 just simply
>>>> vanished completely.
>>>> 
>>>> What on earth?
>>>> 
>>>> Alistair
>>>> 
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>> 
>>>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
>>>> 
>>>>> nup - cleaned out the data dir and restarted the broker. First message in
>>>> vanished. Wasn't persisted. So something is fundamentally broken.
>>>>> 
>>>>> topicA inflightCount = dispatchCount = enqueueCount = 1
>>>>> topicB is completely empty
>>>>> 
>>>>> so the message wasn't persisted, wasn't processed, wasn't routed and just
>>>> vanished from the broker.
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> --------------
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80
>>>>> 
>>>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
>>>>> 
>>>>>> route goes from topicA -> topicB, transacted.
>>>>>> topicA inflightCount = 96 and increases on each batch of incoming
>>>> messages
>>>>>> topicB dispatchCount = enqueueCount
>>>>>> 
>>>>>> wondering if the missing messages are connected to topicA inflightCount.
>>>> I noticed there are two consumers for topicB. The main consumer gets its
>>>> messages fine. Wonder if the second consumer is a durable topic consumer and
>>>> therefore activemq is persisting its messages but it hasn't connected in a
>>>> very long time. Would that cause the topic to get too big? i.e. messages go
>>>> into the topic until the limit is reached. Main consumer pulls messages off
>>>> and messages are able to go onto topicB again. Before consumer pulls and
>>>> after limit reached, messages can't get from topicA -> topicB, hence the
>>>> topicA inflightCount not zero?
>>>>>> 
>>>>>> Alistair
>>>>>> 
>>>>>> --------------
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80
>>>>>> 
>>>>>> On 29 Sep 2011, at 12:17, Tim wrote:
>>>>>> 
>>>>>>> Sorry you might have tried this since I haven't been following this
>>>> thread.
>>>>>>> But can you check your jmx console.
>>>>>>> In particular check 2 things.. the route to see if the number of
>>>> exchanges
>>>>>>> match what you think and how if any exchanges failed.
>>>>>>> Also check the JMX console on activemq for the queue or topic in
>>>> question
>>>>>>> and see how many were enqueued vs dispatched.
>>>>>>> Check your deadletter queue from there too
>>>>>>> 
>>>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>>>>>> <al...@uhi.ac.uk>wrote:
>>>>>>> 
>>>>>>>> dunno - nothing works. Random messages are just vanishing once they
>>>> reach
>>>>>>>> the broker. No trace, no logs, no dead letter queue. Just vanishing.
>>>> I've
>>>>>>>> removed <transacted /> and <process> but it doesn't help. The producer
>>>> is a
>>>>>>>> few secs behind the broker:
>>>>>>>> 
>>>>>>>> sent : 11:25:26
>>>>>>>> arrived : 11:24:57
>>>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
>>>> presumably
>>>>>>>> the timestampplugin doing this
>>>>>>>> message vanishes
>>>>>>>> 
>>>>>>>> but all messages display this clock behaviour and not all vanish.
>>>>>>>> 
>>>>>>>> Alistair
>>>>>>>> 
>>>>>>>> --------------
>>>>>>>> mov eax,1
>>>>>>>> mov ebx,0
>>>>>>>> int 80
>>>>>>>> 
>>>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>>>>>> 
>>>>>>>>> just saw your info about transacted being before from - will change
>>>> that
>>>>>>>> and monitor again.
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> 
>>>>>>>>> Alistair
>>>>>>>>> 
>>>>>>>>> --------------
>>>>>>>>> mov eax,1
>>>>>>>>> mov ebx,0
>>>>>>>>> int 80
>>>>>>>>> 
>>>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>>>>>> 
>>>>>>>>>> just noticed a batch of identical 5 messages, three were missing and
>>>>>>>> another single message vanished. tracer logged nothing. No errors,
>>>> dead
>>>>>>>> letter queue empty.
>>>>>>>>>> 
>>>>>>>>>> One thing that happens is another machine polls the stats topic in
>>>>>>>> activemq every 2mins. Would that cause a problem? It asks for stats on
>>>> the
>>>>>>>> matrix topic, which is part of the transacted route.
>>>>>>>>>> 
>>>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>>>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
>>>> [null]:
>>>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
>>>> vm://matrixBroker#285916
>>>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>>>>>> vm://matrixBroker#285916
>>>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>>>>>>> vm://matrixBroker#285920
>>>>>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>>>>>>> {commandId = 1, responseRequired = true, connectionId =
>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
>>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName =
>>>> null,
>>>>>>>> password = *****, brokerPath = null, brokerMasterConnector = false,
>>>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>>>>>> 
>>>>>>>>>> Alistair
>>>>>>>>>> 
>>>>>>>>>> --------------
>>>>>>>>>> mov eax,1
>>>>>>>>>> mov ebx,0
>>>>>>>>>> int 80
>>>>>>>>>> 
>>>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>>>>>> 
>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>> it is after from - do you mean it should be before?
>>>>>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>>>>>> <transacted />
>>>>>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>>>>>> </route>
>>>>>>>>>>> 
>>>>>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>>>>>> 
>>>>>>>>>>> Alistair
>>>>>>>>>>> 
>>>>>>>>>>> --------------
>>>>>>>>>>> mov eax,1
>>>>>>>>>>> mov ebx,0
>>>>>>>>>>> int 80
>>>>>>>>>>> 
>>>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi
>>>>>>>>>>>> 
>>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>>>>>>> <al...@uhi.ac.uk> wrote:
>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>> no
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> timestamp plugin
>>>>>>>>>>>>> 
>>>>>>>>>>>>> using that
>>>>>>>>>>>>> 
>>>>>>>>>>>>> activemq 5.5.0
>>>>>>>>>>>>> camel 2.8.0
>>>>>>>>>>>>> spring 3.0.5
>>>>>>>>>>>>> 
>>>>>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer is
>>>>>>>> logging so hopefully I can see any errors.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <route id="matrix"
>>>> errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>>>>>> <transacted />
>>>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>>>>>> </route>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <bean id="jmsConnectionFactory"
>>>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>>>>>> depends-on="matrixBrokerID">
>>>>>>>>>>>>>       <property name="brokerURL"
>>>>>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <bean id="jmsTransactionManager"
>>>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>>>>>>       <property name="connectionFactory"
>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <bean id="activemq"
>>>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>>>>>>       <property name="connectionFactory"
>>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>>       <property name="transacted" value="true"/>
>>>>>>>>>>>>>       <property name="transactionManager"
>>>>>>>> ref="jmsTransactionManager"/>
>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>>>>>> <property name="redeliveryPolicy"
>>>>>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>>>>>> </bean>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --------------
>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>> int 80
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>>>>>>> possible.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>>>>>>> And how have you configured the AMQ broker, and the Camel
>>>> context?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
>>>> taariql@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on your
>>>>>>>> logger and
>>>>>>>>>>>>>>> how you configured it.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Here are links to how to enable logging[1] and camel logging
>>>> FAQ[2]
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Taariq
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>>>>>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>>>>>>> camelTracer and
>>>>>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
>>>> tried
>>>>>>>> them all but
>>>>>>>>>>>>>>>> no log appears.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>>> int 80h
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in your
>>>>>>>> route.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>>>>>>> I now have a dead letter channel which is empty after losing
>>>> 9
>>>>>>>> out of 10
>>>>>>>>>>>>>>>> messages. I also added a logging handler which logged nothing.
>>>>>>>> Verified the
>>>>>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>> -----------------
>>>>>>>>>>>> FuseSource
>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>> 
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
getting somewhere.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
      <from uri="activemq:topic:matrix"/>
      <transacted />
      <process ref="matrixProcessor" />
      <to uri="activemq:topic:edirectory"/>
    </route>

- producer clock is 2mins ahead of broker clock, timestampplugin enabled on broker
- no delay between messages at the producer = 62 out of 100 get through the route, consistently
- 3sec delay at producer, around 88 - 97 get through. That's as good as it gets
- 5sec delay at producer, 93 made it through the route
- remove the route and send direct to topic in activemq, no delay, no message loss, consistently

so the messages are being lost in the route. No matter what the delay in sending messages, some are always lost and vanish. There are no errors, nothing in any dead letter queue. They simply vanish. They don't even make it as far as the <process ref="matrixProcessor" />.

sending from a ruby producer that blats them out far quicker than the java producer is even worse. Only 1 - 3 ever get through the route. Removing the route and sending to the activemq topic instead, all messages get through no matter how fast they come.

Alistair

-- 
mov eax,1
mov ebx,0
int 80h




On 29 Sep 2011, at 18:10, Alistair Young wrote:

> no connection pool. What was disturbing was the first message sent to the broker after a restart and clean out of the data dir, disappeared. There's a similar route on the broker that works fine. The only difference is the producer for the wonky route is on windows and is up to 1min ahead of the broker's clock. Would have thought the timestamplugin would take care of that though. Can see it changing the timestamp in the logs.
> 
> Alistair
> 
> --------------
> mov eax,1
> mov ebx,0
> int 80
> 
> On 29 Sep 2011, at 17:31, Tim wrote:
> 
>> Well with only 17 you definitely aren't hitting any prefetch limits or
>> anything like that.
>> Are you using a connection pool?
>> 
>> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>> 
>>> I think this way madness lies.
>>> 
>>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
>>> topicB enqueueCount = 12
>>> 
>>> so 17 came in, 12 made it through, of the 5 that went missing it claims to
>>> have sent 3 to topicB but they never arrived and the last 2 just simply
>>> vanished completely.
>>> 
>>> What on earth?
>>> 
>>> Alistair
>>> 
>>> --------------
>>> mov eax,1
>>> mov ebx,0
>>> int 80
>>> 
>>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
>>> 
>>>> nup - cleaned out the data dir and restarted the broker. First message in
>>> vanished. Wasn't persisted. So something is fundamentally broken.
>>>> 
>>>> topicA inflightCount = dispatchCount = enqueueCount = 1
>>>> topicB is completely empty
>>>> 
>>>> so the message wasn't persisted, wasn't processed, wasn't routed and just
>>> vanished from the broker.
>>>> 
>>>> Alistair
>>>> 
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>> 
>>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
>>>> 
>>>>> route goes from topicA -> topicB, transacted.
>>>>> topicA inflightCount = 96 and increases on each batch of incoming
>>> messages
>>>>> topicB dispatchCount = enqueueCount
>>>>> 
>>>>> wondering if the missing messages are connected to topicA inflightCount.
>>> I noticed there are two consumers for topicB. The main consumer gets its
>>> messages fine. Wonder if the second consumer is a durable topic consumer and
>>> therefore activemq is persisting its messages but it hasn't connected in a
>>> very long time. Would that cause the topic to get too big? i.e. messages go
>>> into the topic until the limit is reached. Main consumer pulls messages off
>>> and messages are able to go onto topicB again. Before consumer pulls and
>>> after limit reached, messages can't get from topicA -> topicB, hence the
>>> topicA inflightCount not zero?
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> --------------
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80
>>>>> 
>>>>> On 29 Sep 2011, at 12:17, Tim wrote:
>>>>> 
>>>>>> Sorry you might have tried this since I haven't been following this
>>> thread.
>>>>>> But can you check your jmx console.
>>>>>> In particular check 2 things.. the route to see if the number of
>>> exchanges
>>>>>> match what you think and how if any exchanges failed.
>>>>>> Also check the JMX console on activemq for the queue or topic in
>>> question
>>>>>> and see how many were enqueued vs dispatched.
>>>>>> Check your deadletter queue from there too
>>>>>> 
>>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>>>>> <al...@uhi.ac.uk>wrote:
>>>>>> 
>>>>>>> dunno - nothing works. Random messages are just vanishing once they
>>> reach
>>>>>>> the broker. No trace, no logs, no dead letter queue. Just vanishing.
>>> I've
>>>>>>> removed <transacted /> and <process> but it doesn't help. The producer
>>> is a
>>>>>>> few secs behind the broker:
>>>>>>> 
>>>>>>> sent : 11:25:26
>>>>>>> arrived : 11:24:57
>>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
>>> presumably
>>>>>>> the timestampplugin doing this
>>>>>>> message vanishes
>>>>>>> 
>>>>>>> but all messages display this clock behaviour and not all vanish.
>>>>>>> 
>>>>>>> Alistair
>>>>>>> 
>>>>>>> --------------
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80
>>>>>>> 
>>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>>>>> 
>>>>>>>> just saw your info about transacted being before from - will change
>>> that
>>>>>>> and monitor again.
>>>>>>>> 
>>>>>>>> thanks,
>>>>>>>> 
>>>>>>>> Alistair
>>>>>>>> 
>>>>>>>> --------------
>>>>>>>> mov eax,1
>>>>>>>> mov ebx,0
>>>>>>>> int 80
>>>>>>>> 
>>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>>>>> 
>>>>>>>>> just noticed a batch of identical 5 messages, three were missing and
>>>>>>> another single message vanished. tracer logged nothing. No errors,
>>> dead
>>>>>>> letter queue empty.
>>>>>>>>> 
>>>>>>>>> One thing that happens is another machine polls the stats topic in
>>>>>>> activemq every 2mins. Would that cause a problem? It asks for stats on
>>> the
>>>>>>> matrix topic, which is part of the transacted route.
>>>>>>>>> 
>>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
>>> [null]:
>>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>>>>> vm://matrixBroker#285916
>>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
>>> vm://matrixBroker#285916
>>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>>>>> vm://matrixBroker#285916
>>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>>>>>> vm://matrixBroker#285920
>>>>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>>>>>> {commandId = 1, responseRequired = true, connectionId =
>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
>>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName =
>>> null,
>>>>>>> password = *****, brokerPath = null, brokerMasterConnector = false,
>>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>>>>> 
>>>>>>>>> Alistair
>>>>>>>>> 
>>>>>>>>> --------------
>>>>>>>>> mov eax,1
>>>>>>>>> mov ebx,0
>>>>>>>>> int 80
>>>>>>>>> 
>>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>>>>> 
>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>> it is after from - do you mean it should be before?
>>>>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>>>>> <transacted />
>>>>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>>>>> </route>
>>>>>>>>>> 
>>>>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>>>>> 
>>>>>>>>>> Alistair
>>>>>>>>>> 
>>>>>>>>>> --------------
>>>>>>>>>> mov eax,1
>>>>>>>>>> mov ebx,0
>>>>>>>>>> int 80
>>>>>>>>>> 
>>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi
>>>>>>>>>>> 
>>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>>>>>> <al...@uhi.ac.uk> wrote:
>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>> no
>>>>>>>>>>>> 
>>>>>>>>>>>>> timestamp plugin
>>>>>>>>>>>> 
>>>>>>>>>>>> using that
>>>>>>>>>>>> 
>>>>>>>>>>>> activemq 5.5.0
>>>>>>>>>>>> camel 2.8.0
>>>>>>>>>>>> spring 3.0.5
>>>>>>>>>>>> 
>>>>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer is
>>>>>>> logging so hopefully I can see any errors.
>>>>>>>>>>>> 
>>>>>>>>>>>> <route id="matrix"
>>> errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>>>>> <transacted />
>>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>>>>> </route>
>>>>>>>>>>>> 
>>>>>>>>>>>> <bean id="jmsConnectionFactory"
>>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>>>>> depends-on="matrixBrokerID">
>>>>>>>>>>>>        <property name="brokerURL"
>>>>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>>>>> </bean>
>>>>>>>>>>>> 
>>>>>>>>>>>> <bean id="jmsTransactionManager"
>>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>>>>>        <property name="connectionFactory"
>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>> </bean>
>>>>>>>>>>>> 
>>>>>>>>>>>> <bean id="activemq"
>>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>>>>>        <property name="connectionFactory"
>>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>>        <property name="transacted" value="true"/>
>>>>>>>>>>>>        <property name="transactionManager"
>>>>>>> ref="jmsTransactionManager"/>
>>>>>>>>>>>> </bean>
>>>>>>>>>>>> 
>>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>>>>> <property name="redeliveryPolicy"
>>>>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>>>>> </bean>
>>>>>>>>>>>> 
>>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>>>>> </bean>
>>>>>>>>>>>> 
>>>>>>>>>>>> thanks,
>>>>>>>>>>>> 
>>>>>>>>>>>> Alistair
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --------------
>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>> int 80
>>>>>>>>>>>> 
>>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>>>>>> possible.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>>>>> 
>>>>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>>>>>> And how have you configured the AMQ broker, and the Camel
>>> context?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
>>> taariql@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on your
>>>>>>> logger and
>>>>>>>>>>>>>> how you configured it.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Here are links to how to enable logging[1] and camel logging
>>> FAQ[2]
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Taariq
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>>>>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>>>>>> camelTracer and
>>>>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
>>> tried
>>>>>>> them all but
>>>>>>>>>>>>>>> no log appears.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>>> int 80h
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in your
>>>>>>> route.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>>>>>> I now have a dead letter channel which is empty after losing
>>> 9
>>>>>>> out of 10
>>>>>>>>>>>>>>> messages. I also added a logging handler which logged nothing.
>>>>>>> Verified the
>>>>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>> FuseSource
>>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>> -----------------
>>>>>>>>>>> FuseSource
>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
no connection pool. What was disturbing was the first message sent to the broker after a restart and clean out of the data dir, disappeared. There's a similar route on the broker that works fine. The only difference is the producer for the wonky route is on windows and is up to 1min ahead of the broker's clock. Would have thought the timestamplugin would take care of that though. Can see it changing the timestamp in the logs.

Alistair

--------------
mov eax,1
mov ebx,0
int 80

On 29 Sep 2011, at 17:31, Tim wrote:

> Well with only 17 you definitely aren't hitting any prefetch limits or
> anything like that.
> Are you using a connection pool?
> 
> On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <al...@uhi.ac.uk>wrote:
> 
>> I think this way madness lies.
>> 
>> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
>> topicB enqueueCount = 12
>> 
>> so 17 came in, 12 made it through, of the 5 that went missing it claims to
>> have sent 3 to topicB but they never arrived and the last 2 just simply
>> vanished completely.
>> 
>> What on earth?
>> 
>> Alistair
>> 
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>> 
>> On 29 Sep 2011, at 15:41, Alistair Young wrote:
>> 
>>> nup - cleaned out the data dir and restarted the broker. First message in
>> vanished. Wasn't persisted. So something is fundamentally broken.
>>> 
>>> topicA inflightCount = dispatchCount = enqueueCount = 1
>>> topicB is completely empty
>>> 
>>> so the message wasn't persisted, wasn't processed, wasn't routed and just
>> vanished from the broker.
>>> 
>>> Alistair
>>> 
>>> --------------
>>> mov eax,1
>>> mov ebx,0
>>> int 80
>>> 
>>> On 29 Sep 2011, at 15:13, Alistair Young wrote:
>>> 
>>>> route goes from topicA -> topicB, transacted.
>>>> topicA inflightCount = 96 and increases on each batch of incoming
>> messages
>>>> topicB dispatchCount = enqueueCount
>>>> 
>>>> wondering if the missing messages are connected to topicA inflightCount.
>> I noticed there are two consumers for topicB. The main consumer gets its
>> messages fine. Wonder if the second consumer is a durable topic consumer and
>> therefore activemq is persisting its messages but it hasn't connected in a
>> very long time. Would that cause the topic to get too big? i.e. messages go
>> into the topic until the limit is reached. Main consumer pulls messages off
>> and messages are able to go onto topicB again. Before consumer pulls and
>> after limit reached, messages can't get from topicA -> topicB, hence the
>> topicA inflightCount not zero?
>>>> 
>>>> Alistair
>>>> 
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>> 
>>>> On 29 Sep 2011, at 12:17, Tim wrote:
>>>> 
>>>>> Sorry you might have tried this since I haven't been following this
>> thread.
>>>>> But can you check your jmx console.
>>>>> In particular check 2 things.. the route to see if the number of
>> exchanges
>>>>> match what you think and how if any exchanges failed.
>>>>> Also check the JMX console on activemq for the queue or topic in
>> question
>>>>> and see how many were enqueued vs dispatched.
>>>>> Check your deadletter queue from there too
>>>>> 
>>>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>>>> <al...@uhi.ac.uk>wrote:
>>>>> 
>>>>>> dunno - nothing works. Random messages are just vanishing once they
>> reach
>>>>>> the broker. No trace, no logs, no dead letter queue. Just vanishing.
>> I've
>>>>>> removed <transacted /> and <process> but it doesn't help. The producer
>> is a
>>>>>> few secs behind the broker:
>>>>>> 
>>>>>> sent : 11:25:26
>>>>>> arrived : 11:24:57
>>>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
>> presumably
>>>>>> the timestampplugin doing this
>>>>>> message vanishes
>>>>>> 
>>>>>> but all messages display this clock behaviour and not all vanish.
>>>>>> 
>>>>>> Alistair
>>>>>> 
>>>>>> --------------
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80
>>>>>> 
>>>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>>>> 
>>>>>>> just saw your info about transacted being before from - will change
>> that
>>>>>> and monitor again.
>>>>>>> 
>>>>>>> thanks,
>>>>>>> 
>>>>>>> Alistair
>>>>>>> 
>>>>>>> --------------
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80
>>>>>>> 
>>>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>>>> 
>>>>>>>> just noticed a batch of identical 5 messages, three were missing and
>>>>>> another single message vanished. tracer logged nothing. No errors,
>> dead
>>>>>> letter queue empty.
>>>>>>>> 
>>>>>>>> One thing that happens is another machine polls the stats topic in
>>>>>> activemq every 2mins. Would that cause a problem? It asks for stats on
>> the
>>>>>> matrix topic, which is part of the transacted route.
>>>>>>>> 
>>>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
>> [null]:
>>>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>>>> vm://matrixBroker#285916
>>>>>>>> 29 September 2011 10:05:07 - Stopped transport:
>> vm://matrixBroker#285916
>>>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>>>> vm://matrixBroker#285916
>>>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>>>>> vm://matrixBroker#285920
>>>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>>>>> {commandId = 1, responseRequired = true, connectionId =
>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
>>>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName =
>> null,
>>>>>> password = *****, brokerPath = null, brokerMasterConnector = false,
>>>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>>>> 
>>>>>>>> Alistair
>>>>>>>> 
>>>>>>>> --------------
>>>>>>>> mov eax,1
>>>>>>>> mov ebx,0
>>>>>>>> int 80
>>>>>>>> 
>>>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>>>> 
>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>> it is after from - do you mean it should be before?
>>>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>>>> <transacted />
>>>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>>>> </route>
>>>>>>>>> 
>>>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>>>> 
>>>>>>>>> Alistair
>>>>>>>>> 
>>>>>>>>> --------------
>>>>>>>>> mov eax,1
>>>>>>>>> mov ebx,0
>>>>>>>>> int 80
>>>>>>>>> 
>>>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>>>> 
>>>>>>>>>> Hi
>>>>>>>>>> 
>>>>>>>>>> <transacted/> Should be after <from>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>>>>> <al...@uhi.ac.uk> wrote:
>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>> no
>>>>>>>>>>> 
>>>>>>>>>>>> timestamp plugin
>>>>>>>>>>> 
>>>>>>>>>>> using that
>>>>>>>>>>> 
>>>>>>>>>>> activemq 5.5.0
>>>>>>>>>>> camel 2.8.0
>>>>>>>>>>> spring 3.0.5
>>>>>>>>>>> 
>>>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer is
>>>>>> logging so hopefully I can see any errors.
>>>>>>>>>>> 
>>>>>>>>>>> <route id="matrix"
>> errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>>>> <transacted />
>>>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>>>> </route>
>>>>>>>>>>> 
>>>>>>>>>>> <bean id="jmsConnectionFactory"
>>>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>>>> depends-on="matrixBrokerID">
>>>>>>>>>>>         <property name="brokerURL"
>>>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>>>> </bean>
>>>>>>>>>>> 
>>>>>>>>>>> <bean id="jmsTransactionManager"
>>>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>>>>         <property name="connectionFactory"
>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>> </bean>
>>>>>>>>>>> 
>>>>>>>>>>> <bean id="activemq"
>>>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>>>>         <property name="connectionFactory"
>>>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>>>         <property name="transacted" value="true"/>
>>>>>>>>>>>         <property name="transactionManager"
>>>>>> ref="jmsTransactionManager"/>
>>>>>>>>>>> </bean>
>>>>>>>>>>> 
>>>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>>>> <property name="redeliveryPolicy"
>>>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>>>> </bean>
>>>>>>>>>>> 
>>>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>>>> </bean>
>>>>>>>>>>> 
>>>>>>>>>>> thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Alistair
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --------------
>>>>>>>>>>> mov eax,1
>>>>>>>>>>> mov ebx,0
>>>>>>>>>>> int 80
>>>>>>>>>>> 
>>>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi
>>>>>>>>>>>> 
>>>>>>>>>>>> Do you use message expiry?
>>>>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>>>>> possible.
>>>>>>>>>>>> 
>>>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>>>> 
>>>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>>>>> And how have you configured the AMQ broker, and the Camel
>> context?
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
>> taariql@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on your
>>>>>> logger and
>>>>>>>>>>>>> how you configured it.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Here are links to how to enable logging[1] and camel logging
>> FAQ[2]
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Taariq
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>>>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>>>>> camelTracer and
>>>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
>> tried
>>>>>> them all but
>>>>>>>>>>>>>> no log appears.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>>>> int 80h
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in your
>>>>>> route.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>>>>> I now have a dead letter channel which is empty after losing
>> 9
>>>>>> out of 10
>>>>>>>>>>>>>> messages. I also added a logging handler which logged nothing.
>>>>>> Verified the
>>>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>> -----------------
>>>>>>>>>>>> FuseSource
>>>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Claus Ibsen
>>>>>>>>>> -----------------
>>>>>>>>>> FuseSource
>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>> 
>> 
>>

Re: Messages being lost from route

Posted by Tim <ch...@gmail.com>.
Well with only 17 you definitely aren't hitting any prefetch limits or
anything like that.
Are you using a connection pool?

On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <al...@uhi.ac.uk>wrote:

> I think this way madness lies.
>
> 17 sent to topicA, dispatchCount = 15, dequeueCount = 12
> topicB enqueueCount = 12
>
> so 17 came in, 12 made it through, of the 5 that went missing it claims to
> have sent 3 to topicB but they never arrived and the last 2 just simply
> vanished completely.
>
> What on earth?
>
> Alistair
>
> --------------
> mov eax,1
> mov ebx,0
> int 80
>
> On 29 Sep 2011, at 15:41, Alistair Young wrote:
>
> > nup - cleaned out the data dir and restarted the broker. First message in
> vanished. Wasn't persisted. So something is fundamentally broken.
> >
> > topicA inflightCount = dispatchCount = enqueueCount = 1
> > topicB is completely empty
> >
> > so the message wasn't persisted, wasn't processed, wasn't routed and just
> vanished from the broker.
> >
> > Alistair
> >
> > --------------
> > mov eax,1
> > mov ebx,0
> > int 80
> >
> > On 29 Sep 2011, at 15:13, Alistair Young wrote:
> >
> >> route goes from topicA -> topicB, transacted.
> >> topicA inflightCount = 96 and increases on each batch of incoming
> messages
> >> topicB dispatchCount = enqueueCount
> >>
> >> wondering if the missing messages are connected to topicA inflightCount.
> I noticed there are two consumers for topicB. The main consumer gets its
> messages fine. Wonder if the second consumer is a durable topic consumer and
> therefore activemq is persisting its messages but it hasn't connected in a
> very long time. Would that cause the topic to get too big? i.e. messages go
> into the topic until the limit is reached. Main consumer pulls messages off
> and messages are able to go onto topicB again. Before consumer pulls and
> after limit reached, messages can't get from topicA -> topicB, hence the
> topicA inflightCount not zero?
> >>
> >> Alistair
> >>
> >> --------------
> >> mov eax,1
> >> mov ebx,0
> >> int 80
> >>
> >> On 29 Sep 2011, at 12:17, Tim wrote:
> >>
> >>> Sorry you might have tried this since I haven't been following this
> thread.
> >>> But can you check your jmx console.
> >>> In particular check 2 things.. the route to see if the number of
> exchanges
> >>> match what you think and how if any exchanges failed.
> >>> Also check the JMX console on activemq for the queue or topic in
> question
> >>> and see how many were enqueued vs dispatched.
> >>> Check your deadletter queue from there too
> >>>
> >>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
> >>> <al...@uhi.ac.uk>wrote:
> >>>
> >>>> dunno - nothing works. Random messages are just vanishing once they
> reach
> >>>> the broker. No trace, no logs, no dead letter queue. Just vanishing.
> I've
> >>>> removed <transacted /> and <process> but it doesn't help. The producer
> is a
> >>>> few secs behind the broker:
> >>>>
> >>>> sent : 11:25:26
> >>>> arrived : 11:24:57
> >>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
> presumably
> >>>> the timestampplugin doing this
> >>>> message vanishes
> >>>>
> >>>> but all messages display this clock behaviour and not all vanish.
> >>>>
> >>>> Alistair
> >>>>
> >>>> --------------
> >>>> mov eax,1
> >>>> mov ebx,0
> >>>> int 80
> >>>>
> >>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
> >>>>
> >>>>> just saw your info about transacted being before from - will change
> that
> >>>> and monitor again.
> >>>>>
> >>>>> thanks,
> >>>>>
> >>>>> Alistair
> >>>>>
> >>>>> --------------
> >>>>> mov eax,1
> >>>>> mov ebx,0
> >>>>> int 80
> >>>>>
> >>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
> >>>>>
> >>>>>> just noticed a batch of identical 5 messages, three were missing and
> >>>> another single message vanished. tracer logged nothing. No errors,
> dead
> >>>> letter queue empty.
> >>>>>>
> >>>>>> One thing that happens is another machine polls the stats topic in
> >>>> activemq every 2mins. Would that cause a problem? It asks for stats on
> the
> >>>> matrix topic, which is part of the transacted route.
> >>>>>>
> >>>>>> 29 September 2011 10:05:07 - Adding destination :
> >>>> Topic:ActiveMQ.Advisory.Connection
> >>>>>> 29 September 2011 10:05:07 - Creating new transaction with name
> [null]:
> >>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
> >>>>>> 29 September 2011 10:05:07 - Stopping connection:
> >>>> vm://matrixBroker#285916
> >>>>>> 29 September 2011 10:05:07 - Stopped transport:
> vm://matrixBroker#285916
> >>>>>> 29 September 2011 10:05:07 - Connection Stopped:
> >>>> vm://matrixBroker#285916
> >>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
> >>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
> >>>> vm://matrixBroker#285920
> >>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
> >>>> {commandId = 1, responseRequired = true, connectionId =
> >>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
> >>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName =
> null,
> >>>> password = *****, brokerPath = null, brokerMasterConnector = false,
> >>>> manageable = true, clientMaster = true, faultTolerant = false}
> >>>>>>
> >>>>>> Alistair
> >>>>>>
> >>>>>> --------------
> >>>>>> mov eax,1
> >>>>>> mov ebx,0
> >>>>>> int 80
> >>>>>>
> >>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
> >>>>>>
> >>>>>>>> <transacted/> Should be after <from>
> >>>>>>> it is after from - do you mean it should be before?
> >>>>>>> <route id="eDirSuccessBroadcast">
> >>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
> >>>>>>> <transacted />
> >>>>>>> <process ref="groupwiseProcessor" />
> >>>>>>> <to uri="activemq:topic:blackboard"/>
> >>>>>>> </route>
> >>>>>>>
> >>>>>>> thanks for the dead letter tips, will apply them.
> >>>>>>>
> >>>>>>> Alistair
> >>>>>>>
> >>>>>>> --------------
> >>>>>>> mov eax,1
> >>>>>>> mov ebx,0
> >>>>>>> int 80
> >>>>>>>
> >>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
> >>>>>>>
> >>>>>>>> Hi
> >>>>>>>>
> >>>>>>>> <transacted/> Should be after <from>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
> >>>>>>>> <al...@uhi.ac.uk> wrote:
> >>>>>>>>>> Do you use message expiry?
> >>>>>>>>> no
> >>>>>>>>>
> >>>>>>>>>> timestamp plugin
> >>>>>>>>>
> >>>>>>>>> using that
> >>>>>>>>>
> >>>>>>>>> activemq 5.5.0
> >>>>>>>>> camel 2.8.0
> >>>>>>>>> spring 3.0.5
> >>>>>>>>>
> >>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer is
> >>>> logging so hopefully I can see any errors.
> >>>>>>>>>
> >>>>>>>>> <route id="matrix"
> errorHandlerRef="matrixDeadLetterErrorHandler">
> >>>>>>>>> <from uri="activemq:topic:matrix"/>
> >>>>>>>>> <process ref="matrixProcessor" />
> >>>>>>>>> <transacted />
> >>>>>>>>> <to uri="activemq:topic:edirectory"/>
> >>>>>>>>> </route>
> >>>>>>>>>
> >>>>>>>>>  <bean id="jmsConnectionFactory"
> >>>> class="org.apache.activemq.ActiveMQConnectionFactory"
> >>>> depends-on="matrixBrokerID">
> >>>>>>>>>          <property name="brokerURL"
> >>>> value="vm://matrixBroker?create=false"/>
> >>>>>>>>>  </bean>
> >>>>>>>>>
> >>>>>>>>>  <bean id="jmsTransactionManager"
> >>>> class="org.springframework.jms.connection.JmsTransactionManager">
> >>>>>>>>>          <property name="connectionFactory"
> >>>> ref="jmsConnectionFactory"/>
> >>>>>>>>>  </bean>
> >>>>>>>>>
> >>>>>>>>>  <bean id="activemq"
> >>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
> >>>>>>>>>          <property name="connectionFactory"
> >>>> ref="jmsConnectionFactory"/>
> >>>>>>>>>          <property name="transacted" value="true"/>
> >>>>>>>>>          <property name="transactionManager"
> >>>> ref="jmsTransactionManager"/>
> >>>>>>>>>  </bean>
> >>>>>>>>>
> >>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
> >>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
> >>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
> >>>>>>>>> <property name="redeliveryPolicy"
> >>>> ref="matrixRedeliveryPolicyConfig"/>
> >>>>>>>>> </bean>
> >>>>>>>>>
> >>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
> >>>> class="org.apache.camel.processor.RedeliveryPolicy">
> >>>>>>>>> <property name="maximumRedeliveries" value="10"/>
> >>>>>>>>> <property name="redeliveryDelay" value="250"/>
> >>>>>>>>> </bean>
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>>
> >>>>>>>>> Alistair
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --------------
> >>>>>>>>> mov eax,1
> >>>>>>>>> mov ebx,0
> >>>>>>>>> int 80
> >>>>>>>>>
> >>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi
> >>>>>>>>>>
> >>>>>>>>>> Do you use message expiry?
> >>>>>>>>>> Make sure clocks between server/clients is synced as much as
> >>>> possible.
> >>>>>>>>>>
> >>>>>>>>>> There is a timestamp plugin
> >>>>>>>>>> http://activemq.apache.org/timestampplugin.html
> >>>>>>>>>>
> >>>>>>>>>> And do you use queue or topic.
> >>>>>>>>>> What version of AMQ and Camel are you using?
> >>>>>>>>>> And how have you configured the AMQ broker, and the Camel
> context?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
> taariql@gmail.com>
> >>>> wrote:
> >>>>>>>>>>> Hi
> >>>>>>>>>>>
> >>>>>>>>>>> Where the logs go, if it's logged at all, still depends on your
> >>>> logger and
> >>>>>>>>>>> how you configured it.
> >>>>>>>>>>>
> >>>>>>>>>>> Here are links to how to enable logging[1] and camel logging
> FAQ[2]
> >>>>>>>>>>>
> >>>>>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
> >>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
> >>>>>>>>>>>
> >>>>>>>>>>> Taariq
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
> >>>> alistair.young@uhi.ac.uk>wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> which is the best trace method to use? trace="true", or
> >>>> camelTracer and
> >>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've
> tried
> >>>> them all but
> >>>>>>>>>>>> no log appears.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Alistair
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> mov eax,1
> >>>>>>>>>>>> mov ebx,0
> >>>>>>>>>>>> int 80h
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in your
> >>>> route.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> regards, Marco
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
> >>>>>>>>>>>>>> I now have a dead letter channel which is empty after losing
> 9
> >>>> out of 10
> >>>>>>>>>>>> messages. I also added a logging handler which logged nothing.
> >>>> Verified the
> >>>>>>>>>>>> messages arrived at the broker, then they just vanished.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Alistair
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Claus Ibsen
> >>>>>>>>>> -----------------
> >>>>>>>>>> FuseSource
> >>>>>>>>>> Email: cibsen@fusesource.com
> >>>>>>>>>> Web: http://fusesource.com
> >>>>>>>>>> Twitter: davsclaus, fusenews
> >>>>>>>>>> Blog: http://davsclaus.blogspot.com/
> >>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Claus Ibsen
> >>>>>>>> -----------------
> >>>>>>>> FuseSource
> >>>>>>>> Email: cibsen@fusesource.com
> >>>>>>>> Web: http://fusesource.com
> >>>>>>>> Twitter: davsclaus, fusenews
> >>>>>>>> Blog: http://davsclaus.blogspot.com/
> >>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>
> >
>
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
I think this way madness lies.

17 sent to topicA, dispatchCount = 15, dequeueCount = 12
topicB enqueueCount = 12

so 17 came in, 12 made it through, of the 5 that went missing it claims to have sent 3 to topicB but they never arrived and the last 2 just simply vanished completely.

What on earth?

Alistair

--------------
mov eax,1
mov ebx,0
int 80

On 29 Sep 2011, at 15:41, Alistair Young wrote:

> nup - cleaned out the data dir and restarted the broker. First message in vanished. Wasn't persisted. So something is fundamentally broken.
> 
> topicA inflightCount = dispatchCount = enqueueCount = 1
> topicB is completely empty
> 
> so the message wasn't persisted, wasn't processed, wasn't routed and just vanished from the broker.
> 
> Alistair
> 
> --------------
> mov eax,1
> mov ebx,0
> int 80
> 
> On 29 Sep 2011, at 15:13, Alistair Young wrote:
> 
>> route goes from topicA -> topicB, transacted.
>> topicA inflightCount = 96 and increases on each batch of incoming messages
>> topicB dispatchCount = enqueueCount
>> 
>> wondering if the missing messages are connected to topicA inflightCount. I noticed there are two consumers for topicB. The main consumer gets its messages fine. Wonder if the second consumer is a durable topic consumer and therefore activemq is persisting its messages but it hasn't connected in a very long time. Would that cause the topic to get too big? i.e. messages go into the topic until the limit is reached. Main consumer pulls messages off and messages are able to go onto topicB again. Before consumer pulls and after limit reached, messages can't get from topicA -> topicB, hence the topicA inflightCount not zero?
>> 
>> Alistair
>> 
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>> 
>> On 29 Sep 2011, at 12:17, Tim wrote:
>> 
>>> Sorry you might have tried this since I haven't been following this thread.
>>> But can you check your jmx console.
>>> In particular check 2 things.. the route to see if the number of exchanges
>>> match what you think and how if any exchanges failed.
>>> Also check the JMX console on activemq for the queue or topic in question
>>> and see how many were enqueued vs dispatched.
>>> Check your deadletter queue from there too
>>> 
>>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>>> <al...@uhi.ac.uk>wrote:
>>> 
>>>> dunno - nothing works. Random messages are just vanishing once they reach
>>>> the broker. No trace, no logs, no dead letter queue. Just vanishing. I've
>>>> removed <transacted /> and <process> but it doesn't help. The producer is a
>>>> few secs behind the broker:
>>>> 
>>>> sent : 11:25:26
>>>> arrived : 11:24:57
>>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably
>>>> the timestampplugin doing this
>>>> message vanishes
>>>> 
>>>> but all messages display this clock behaviour and not all vanish.
>>>> 
>>>> Alistair
>>>> 
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>> 
>>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>>> 
>>>>> just saw your info about transacted being before from - will change that
>>>> and monitor again.
>>>>> 
>>>>> thanks,
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> --------------
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80
>>>>> 
>>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>>> 
>>>>>> just noticed a batch of identical 5 messages, three were missing and
>>>> another single message vanished. tracer logged nothing. No errors, dead
>>>> letter queue empty.
>>>>>> 
>>>>>> One thing that happens is another machine polls the stats topic in
>>>> activemq every 2mins. Would that cause a problem? It asks for stats on the
>>>> matrix topic, which is part of the transacted route.
>>>>>> 
>>>>>> 29 September 2011 10:05:07 - Adding destination :
>>>> Topic:ActiveMQ.Advisory.Connection
>>>>>> 29 September 2011 10:05:07 - Creating new transaction with name [null]:
>>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>>> vm://matrixBroker#285916
>>>>>> 29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
>>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>>> vm://matrixBroker#285916
>>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>>> vm://matrixBroker#285920
>>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>>> {commandId = 1, responseRequired = true, connectionId =
>>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
>>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null,
>>>> password = *****, brokerPath = null, brokerMasterConnector = false,
>>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>>> 
>>>>>> Alistair
>>>>>> 
>>>>>> --------------
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80
>>>>>> 
>>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>>> 
>>>>>>>> <transacted/> Should be after <from>
>>>>>>> it is after from - do you mean it should be before?
>>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>> <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>> <transacted />
>>>>>>> <process ref="groupwiseProcessor" />
>>>>>>> <to uri="activemq:topic:blackboard"/>
>>>>>>> </route>
>>>>>>> 
>>>>>>> thanks for the dead letter tips, will apply them.
>>>>>>> 
>>>>>>> Alistair
>>>>>>> 
>>>>>>> --------------
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80
>>>>>>> 
>>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>>> 
>>>>>>>> Hi
>>>>>>>> 
>>>>>>>> <transacted/> Should be after <from>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>>> <al...@uhi.ac.uk> wrote:
>>>>>>>>>> Do you use message expiry?
>>>>>>>>> no
>>>>>>>>> 
>>>>>>>>>> timestamp plugin
>>>>>>>>> 
>>>>>>>>> using that
>>>>>>>>> 
>>>>>>>>> activemq 5.5.0
>>>>>>>>> camel 2.8.0
>>>>>>>>> spring 3.0.5
>>>>>>>>> 
>>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer is
>>>> logging so hopefully I can see any errors.
>>>>>>>>> 
>>>>>>>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>>> <transacted />
>>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>>> </route>
>>>>>>>>> 
>>>>>>>>>  <bean id="jmsConnectionFactory"
>>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>>> depends-on="matrixBrokerID">
>>>>>>>>>          <property name="brokerURL"
>>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>>  </bean>
>>>>>>>>> 
>>>>>>>>>  <bean id="jmsTransactionManager"
>>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>>          <property name="connectionFactory"
>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>  </bean>
>>>>>>>>> 
>>>>>>>>>  <bean id="activemq"
>>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>>          <property name="connectionFactory"
>>>> ref="jmsConnectionFactory"/>
>>>>>>>>>          <property name="transacted" value="true"/>
>>>>>>>>>          <property name="transactionManager"
>>>> ref="jmsTransactionManager"/>
>>>>>>>>>  </bean>
>>>>>>>>> 
>>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>>> <property name="redeliveryPolicy"
>>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>>> </bean>
>>>>>>>>> 
>>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>>> </bean>
>>>>>>>>> 
>>>>>>>>> thanks,
>>>>>>>>> 
>>>>>>>>> Alistair
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --------------
>>>>>>>>> mov eax,1
>>>>>>>>> mov ebx,0
>>>>>>>>> int 80
>>>>>>>>> 
>>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>>> 
>>>>>>>>>> Hi
>>>>>>>>>> 
>>>>>>>>>> Do you use message expiry?
>>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>>> possible.
>>>>>>>>>> 
>>>>>>>>>> There is a timestamp plugin
>>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>>> 
>>>>>>>>>> And do you use queue or topic.
>>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>>> And how have you configured the AMQ broker, and the Camel context?
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com>
>>>> wrote:
>>>>>>>>>>> Hi
>>>>>>>>>>> 
>>>>>>>>>>> Where the logs go, if it's logged at all, still depends on your
>>>> logger and
>>>>>>>>>>> how you configured it.
>>>>>>>>>>> 
>>>>>>>>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>>>>>>>>> 
>>>>>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>>> 
>>>>>>>>>>> Taariq
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>>> camelTracer and
>>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've tried
>>>> them all but
>>>>>>>>>>>> no log appears.
>>>>>>>>>>>> 
>>>>>>>>>>>> Alistair
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> mov eax,1
>>>>>>>>>>>> mov ebx,0
>>>>>>>>>>>> int 80h
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in your
>>>> route.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>>> I now have a dead letter channel which is empty after losing 9
>>>> out of 10
>>>>>>>>>>>> messages. I also added a logging handler which logged nothing.
>>>> Verified the
>>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Claus Ibsen
>>>>>>>>>> -----------------
>>>>>>>>>> FuseSource
>>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>>> Web: http://fusesource.com
>>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> -----------------
>>>>>>>> FuseSource
>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>> Web: http://fusesource.com
>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>> 
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
nup - cleaned out the data dir and restarted the broker. First message in vanished. Wasn't persisted. So something is fundamentally broken.

topicA inflightCount = dispatchCount = enqueueCount = 1
topicB is completely empty

so the message wasn't persisted, wasn't processed, wasn't routed and just vanished from the broker.

Alistair

--------------
mov eax,1
mov ebx,0
int 80

On 29 Sep 2011, at 15:13, Alistair Young wrote:

> route goes from topicA -> topicB, transacted.
> topicA inflightCount = 96 and increases on each batch of incoming messages
> topicB dispatchCount = enqueueCount
> 
> wondering if the missing messages are connected to topicA inflightCount. I noticed there are two consumers for topicB. The main consumer gets its messages fine. Wonder if the second consumer is a durable topic consumer and therefore activemq is persisting its messages but it hasn't connected in a very long time. Would that cause the topic to get too big? i.e. messages go into the topic until the limit is reached. Main consumer pulls messages off and messages are able to go onto topicB again. Before consumer pulls and after limit reached, messages can't get from topicA -> topicB, hence the topicA inflightCount not zero?
> 
> Alistair
> 
> --------------
> mov eax,1
> mov ebx,0
> int 80
> 
> On 29 Sep 2011, at 12:17, Tim wrote:
> 
>> Sorry you might have tried this since I haven't been following this thread.
>> But can you check your jmx console.
>> In particular check 2 things.. the route to see if the number of exchanges
>> match what you think and how if any exchanges failed.
>> Also check the JMX console on activemq for the queue or topic in question
>> and see how many were enqueued vs dispatched.
>> Check your deadletter queue from there too
>> 
>> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
>> <al...@uhi.ac.uk>wrote:
>> 
>>> dunno - nothing works. Random messages are just vanishing once they reach
>>> the broker. No trace, no logs, no dead letter queue. Just vanishing. I've
>>> removed <transacted /> and <process> but it doesn't help. The producer is a
>>> few secs behind the broker:
>>> 
>>> sent : 11:25:26
>>> arrived : 11:24:57
>>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably
>>> the timestampplugin doing this
>>> message vanishes
>>> 
>>> but all messages display this clock behaviour and not all vanish.
>>> 
>>> Alistair
>>> 
>>> --------------
>>> mov eax,1
>>> mov ebx,0
>>> int 80
>>> 
>>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>>> 
>>>> just saw your info about transacted being before from - will change that
>>> and monitor again.
>>>> 
>>>> thanks,
>>>> 
>>>> Alistair
>>>> 
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>> 
>>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>>> 
>>>>> just noticed a batch of identical 5 messages, three were missing and
>>> another single message vanished. tracer logged nothing. No errors, dead
>>> letter queue empty.
>>>>> 
>>>>> One thing that happens is another machine polls the stats topic in
>>> activemq every 2mins. Would that cause a problem? It asks for stats on the
>>> matrix topic, which is part of the transacted route.
>>>>> 
>>>>> 29 September 2011 10:05:07 - Adding destination :
>>> Topic:ActiveMQ.Advisory.Connection
>>>>> 29 September 2011 10:05:07 - Creating new transaction with name [null]:
>>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>>> 29 September 2011 10:05:07 - Stopping connection:
>>> vm://matrixBroker#285916
>>>>> 29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
>>>>> 29 September 2011 10:05:07 - Connection Stopped:
>>> vm://matrixBroker#285916
>>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>>> vm://matrixBroker#285920
>>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>>> {commandId = 1, responseRequired = true, connectionId =
>>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
>>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null,
>>> password = *****, brokerPath = null, brokerMasterConnector = false,
>>> manageable = true, clientMaster = true, faultTolerant = false}
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> --------------
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80
>>>>> 
>>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>>> 
>>>>>>> <transacted/> Should be after <from>
>>>>>> it is after from - do you mean it should be before?
>>>>>> <route id="eDirSuccessBroadcast">
>>>>>>  <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>>  <transacted />
>>>>>>  <process ref="groupwiseProcessor" />
>>>>>>  <to uri="activemq:topic:blackboard"/>
>>>>>> </route>
>>>>>> 
>>>>>> thanks for the dead letter tips, will apply them.
>>>>>> 
>>>>>> Alistair
>>>>>> 
>>>>>> --------------
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80
>>>>>> 
>>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>>> 
>>>>>>> Hi
>>>>>>> 
>>>>>>> <transacted/> Should be after <from>
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>>> <al...@uhi.ac.uk> wrote:
>>>>>>>>> Do you use message expiry?
>>>>>>>> no
>>>>>>>> 
>>>>>>>>> timestamp plugin
>>>>>>>> 
>>>>>>>> using that
>>>>>>>> 
>>>>>>>> activemq 5.5.0
>>>>>>>> camel 2.8.0
>>>>>>>> spring 3.0.5
>>>>>>>> 
>>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer is
>>> logging so hopefully I can see any errors.
>>>>>>>> 
>>>>>>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>> <from uri="activemq:topic:matrix"/>
>>>>>>>> <process ref="matrixProcessor" />
>>>>>>>> <transacted />
>>>>>>>> <to uri="activemq:topic:edirectory"/>
>>>>>>>> </route>
>>>>>>>> 
>>>>>>>>   <bean id="jmsConnectionFactory"
>>> class="org.apache.activemq.ActiveMQConnectionFactory"
>>> depends-on="matrixBrokerID">
>>>>>>>>           <property name="brokerURL"
>>> value="vm://matrixBroker?create=false"/>
>>>>>>>>   </bean>
>>>>>>>> 
>>>>>>>>   <bean id="jmsTransactionManager"
>>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>>           <property name="connectionFactory"
>>> ref="jmsConnectionFactory"/>
>>>>>>>>   </bean>
>>>>>>>> 
>>>>>>>>   <bean id="activemq"
>>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>>           <property name="connectionFactory"
>>> ref="jmsConnectionFactory"/>
>>>>>>>>           <property name="transacted" value="true"/>
>>>>>>>>           <property name="transactionManager"
>>> ref="jmsTransactionManager"/>
>>>>>>>>   </bean>
>>>>>>>> 
>>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>>> <property name="redeliveryPolicy"
>>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>>> </bean>
>>>>>>>> 
>>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>>> </bean>
>>>>>>>> 
>>>>>>>> thanks,
>>>>>>>> 
>>>>>>>> Alistair
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --------------
>>>>>>>> mov eax,1
>>>>>>>> mov ebx,0
>>>>>>>> int 80
>>>>>>>> 
>>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>>> 
>>>>>>>>> Hi
>>>>>>>>> 
>>>>>>>>> Do you use message expiry?
>>>>>>>>> Make sure clocks between server/clients is synced as much as
>>> possible.
>>>>>>>>> 
>>>>>>>>> There is a timestamp plugin
>>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>>> 
>>>>>>>>> And do you use queue or topic.
>>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>>> And how have you configured the AMQ broker, and the Camel context?
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com>
>>> wrote:
>>>>>>>>>> Hi
>>>>>>>>>> 
>>>>>>>>>> Where the logs go, if it's logged at all, still depends on your
>>> logger and
>>>>>>>>>> how you configured it.
>>>>>>>>>> 
>>>>>>>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>>>>>>>> 
>>>>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>>> 
>>>>>>>>>> Taariq
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>>>> 
>>>>>>>>>>> which is the best trace method to use? trace="true", or
>>> camelTracer and
>>>>>>>>>>> traceFormatter beans? and where does the log end up? I've tried
>>> them all but
>>>>>>>>>>> no log appears.
>>>>>>>>>>> 
>>>>>>>>>>> Alistair
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> mov eax,1
>>>>>>>>>>> mov ebx,0
>>>>>>>>>>> int 80h
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> I suggest enable tracing to see exactly what happens in your
>>> route.
>>>>>>>>>>>> 
>>>>>>>>>>>> regards, Marco
>>>>>>>>>>>> 
>>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>>> I now have a dead letter channel which is empty after losing 9
>>> out of 10
>>>>>>>>>>> messages. I also added a logging handler which logged nothing.
>>> Verified the
>>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Alistair
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Claus Ibsen
>>>>>>>>> -----------------
>>>>>>>>> FuseSource
>>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>>> Web: http://fusesource.com
>>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> -----------------
>>>>>>> FuseSource
>>>>>>> Email: cibsen@fusesource.com
>>>>>>> Web: http://fusesource.com
>>>>>>> Twitter: davsclaus, fusenews
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
route goes from topicA -> topicB, transacted.
topicA inflightCount = 96 and increases on each batch of incoming messages
topicB dispatchCount = enqueueCount

wondering if the missing messages are connected to topicA inflightCount. I noticed there are two consumers for topicB. The main consumer gets its messages fine. Wonder if the second consumer is a durable topic consumer and therefore activemq is persisting its messages but it hasn't connected in a very long time. Would that cause the topic to get too big? i.e. messages go into the topic until the limit is reached. Main consumer pulls messages off and messages are able to go onto topicB again. Before consumer pulls and after limit reached, messages can't get from topicA -> topicB, hence the topicA inflightCount not zero?

Alistair

--------------
mov eax,1
mov ebx,0
int 80

On 29 Sep 2011, at 12:17, Tim wrote:

> Sorry you might have tried this since I haven't been following this thread.
> But can you check your jmx console.
> In particular check 2 things.. the route to see if the number of exchanges
> match what you think and how if any exchanges failed.
> Also check the JMX console on activemq for the queue or topic in question
> and see how many were enqueued vs dispatched.
> Check your deadletter queue from there too
> 
> On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
> <al...@uhi.ac.uk>wrote:
> 
>> dunno - nothing works. Random messages are just vanishing once they reach
>> the broker. No trace, no logs, no dead letter queue. Just vanishing. I've
>> removed <transacted /> and <process> but it doesn't help. The producer is a
>> few secs behind the broker:
>> 
>> sent : 11:25:26
>> arrived : 11:24:57
>> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably
>> the timestampplugin doing this
>> message vanishes
>> 
>> but all messages display this clock behaviour and not all vanish.
>> 
>> Alistair
>> 
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>> 
>> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>> 
>>> just saw your info about transacted being before from - will change that
>> and monitor again.
>>> 
>>> thanks,
>>> 
>>> Alistair
>>> 
>>> --------------
>>> mov eax,1
>>> mov ebx,0
>>> int 80
>>> 
>>> On 29 Sep 2011, at 10:18, Alistair Young wrote:
>>> 
>>>> just noticed a batch of identical 5 messages, three were missing and
>> another single message vanished. tracer logged nothing. No errors, dead
>> letter queue empty.
>>>> 
>>>> One thing that happens is another machine polls the stats topic in
>> activemq every 2mins. Would that cause a problem? It asks for stats on the
>> matrix topic, which is part of the transacted route.
>>>> 
>>>> 29 September 2011 10:05:07 - Adding destination :
>> Topic:ActiveMQ.Advisory.Connection
>>>> 29 September 2011 10:05:07 - Creating new transaction with name [null]:
>> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>>>> 29 September 2011 10:05:07 - Stopping connection:
>> vm://matrixBroker#285916
>>>> 29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
>>>> 29 September 2011 10:05:07 - Connection Stopped:
>> vm://matrixBroker#285916
>>>> 29 September 2011 10:05:07 - Setting up new connection id:
>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
>> vm://matrixBroker#285920
>>>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
>> {commandId = 1, responseRequired = true, connectionId =
>> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
>> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null,
>> password = *****, brokerPath = null, brokerMasterConnector = false,
>> manageable = true, clientMaster = true, faultTolerant = false}
>>>> 
>>>> Alistair
>>>> 
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>> 
>>>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>>>> 
>>>>>> <transacted/> Should be after <from>
>>>>> it is after from - do you mean it should be before?
>>>>> <route id="eDirSuccessBroadcast">
>>>>>   <from uri="activemq:topic:edirectoryprocessed"/>
>>>>>   <transacted />
>>>>>   <process ref="groupwiseProcessor" />
>>>>>   <to uri="activemq:topic:blackboard"/>
>>>>> </route>
>>>>> 
>>>>> thanks for the dead letter tips, will apply them.
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> --------------
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80
>>>>> 
>>>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>>>> 
>>>>>> Hi
>>>>>> 
>>>>>> <transacted/> Should be after <from>
>>>>>> 
>>>>>> 
>>>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>>>> <al...@uhi.ac.uk> wrote:
>>>>>>>> Do you use message expiry?
>>>>>>> no
>>>>>>> 
>>>>>>>> timestamp plugin
>>>>>>> 
>>>>>>> using that
>>>>>>> 
>>>>>>> activemq 5.5.0
>>>>>>> camel 2.8.0
>>>>>>> spring 3.0.5
>>>>>>> 
>>>>>>> noticed sl4j errors on startup, fixed that and now the tracer is
>> logging so hopefully I can see any errors.
>>>>>>> 
>>>>>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>>>  <from uri="activemq:topic:matrix"/>
>>>>>>>  <process ref="matrixProcessor" />
>>>>>>>  <transacted />
>>>>>>>  <to uri="activemq:topic:edirectory"/>
>>>>>>> </route>
>>>>>>> 
>>>>>>>    <bean id="jmsConnectionFactory"
>> class="org.apache.activemq.ActiveMQConnectionFactory"
>> depends-on="matrixBrokerID">
>>>>>>>            <property name="brokerURL"
>> value="vm://matrixBroker?create=false"/>
>>>>>>>    </bean>
>>>>>>> 
>>>>>>>    <bean id="jmsTransactionManager"
>> class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>>>            <property name="connectionFactory"
>> ref="jmsConnectionFactory"/>
>>>>>>>    </bean>
>>>>>>> 
>>>>>>>    <bean id="activemq"
>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>>>            <property name="connectionFactory"
>> ref="jmsConnectionFactory"/>
>>>>>>>            <property name="transacted" value="true"/>
>>>>>>>            <property name="transactionManager"
>> ref="jmsTransactionManager"/>
>>>>>>>    </bean>
>>>>>>> 
>>>>>>> <bean id="matrixDeadLetterErrorHandler"
>> class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>>>> <property name="redeliveryPolicy"
>> ref="matrixRedeliveryPolicyConfig"/>
>>>>>>> </bean>
>>>>>>> 
>>>>>>> <bean id="matrixRedeliveryPolicyConfig"
>> class="org.apache.camel.processor.RedeliveryPolicy">
>>>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>>>> <property name="redeliveryDelay" value="250"/>
>>>>>>> </bean>
>>>>>>> 
>>>>>>> thanks,
>>>>>>> 
>>>>>>> Alistair
>>>>>>> 
>>>>>>> 
>>>>>>> --------------
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80
>>>>>>> 
>>>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>>>> 
>>>>>>>> Hi
>>>>>>>> 
>>>>>>>> Do you use message expiry?
>>>>>>>> Make sure clocks between server/clients is synced as much as
>> possible.
>>>>>>>> 
>>>>>>>> There is a timestamp plugin
>>>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>>>> 
>>>>>>>> And do you use queue or topic.
>>>>>>>> What version of AMQ and Camel are you using?
>>>>>>>> And how have you configured the AMQ broker, and the Camel context?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com>
>> wrote:
>>>>>>>>> Hi
>>>>>>>>> 
>>>>>>>>> Where the logs go, if it's logged at all, still depends on your
>> logger and
>>>>>>>>> how you configured it.
>>>>>>>>> 
>>>>>>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>>>>>>> 
>>>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>>>> 
>>>>>>>>> Taariq
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
>> alistair.young@uhi.ac.uk>wrote:
>>>>>>>>> 
>>>>>>>>>> which is the best trace method to use? trace="true", or
>> camelTracer and
>>>>>>>>>> traceFormatter beans? and where does the log end up? I've tried
>> them all but
>>>>>>>>>> no log appears.
>>>>>>>>>> 
>>>>>>>>>> Alistair
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> mov eax,1
>>>>>>>>>> mov ebx,0
>>>>>>>>>> int 80h
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> I suggest enable tracing to see exactly what happens in your
>> route.
>>>>>>>>>>> 
>>>>>>>>>>> regards, Marco
>>>>>>>>>>> 
>>>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>>>> I now have a dead letter channel which is empty after losing 9
>> out of 10
>>>>>>>>>> messages. I also added a logging handler which logged nothing.
>> Verified the
>>>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>>>> 
>>>>>>>>>>>> Alistair
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> -----------------
>>>>>>>> FuseSource
>>>>>>>> Email: cibsen@fusesource.com
>>>>>>>> Web: http://fusesource.com
>>>>>>>> Twitter: davsclaus, fusenews
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> -----------------
>>>>>> FuseSource
>>>>>> Email: cibsen@fusesource.com
>>>>>> Web: http://fusesource.com
>>>>>> Twitter: davsclaus, fusenews
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> 
>>>> 
>>> 
>> 
>>

Re: Messages being lost from route

Posted by Tim <ch...@gmail.com>.
Sorry you might have tried this since I haven't been following this thread.
But can you check your jmx console.
In particular check 2 things.. the route to see if the number of exchanges
match what you think and how if any exchanges failed.
Also check the JMX console on activemq for the queue or topic in question
and see how many were enqueued vs dispatched.
Check your deadletter queue from there too

On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
<al...@uhi.ac.uk>wrote:

> dunno - nothing works. Random messages are just vanishing once they reach
> the broker. No trace, no logs, no dead letter queue. Just vanishing. I've
> removed <transacted /> and <process> but it doesn't help. The producer is a
> few secs behind the broker:
>
> sent : 11:25:26
> arrived : 11:24:57
> timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably
> the timestampplugin doing this
> message vanishes
>
> but all messages display this clock behaviour and not all vanish.
>
> Alistair
>
> --------------
> mov eax,1
> mov ebx,0
> int 80
>
> On 29 Sep 2011, at 10:24, Alistair Young wrote:
>
> > just saw your info about transacted being before from - will change that
> and monitor again.
> >
> > thanks,
> >
> > Alistair
> >
> > --------------
> > mov eax,1
> > mov ebx,0
> > int 80
> >
> > On 29 Sep 2011, at 10:18, Alistair Young wrote:
> >
> >> just noticed a batch of identical 5 messages, three were missing and
> another single message vanished. tracer logged nothing. No errors, dead
> letter queue empty.
> >>
> >> One thing that happens is another machine polls the stats topic in
> activemq every 2mins. Would that cause a problem? It asks for stats on the
> matrix topic, which is part of the transacted route.
> >>
> >> 29 September 2011 10:05:07 - Adding destination :
> Topic:ActiveMQ.Advisory.Connection
> >> 29 September 2011 10:05:07 - Creating new transaction with name [null]:
> PROPAGATION_REQUIRED,ISOLATION_DEFAULT
> >> 29 September 2011 10:05:07 - Stopping connection:
> vm://matrixBroker#285916
> >> 29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
> >> 29 September 2011 10:05:07 - Connection Stopped:
> vm://matrixBroker#285916
> >> 29 September 2011 10:05:07 - Setting up new connection id:
> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
> vm://matrixBroker#285920
> >> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
> {commandId = 1, responseRequired = true, connectionId =
> ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
> ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null,
> password = *****, brokerPath = null, brokerMasterConnector = false,
> manageable = true, clientMaster = true, faultTolerant = false}
> >>
> >> Alistair
> >>
> >> --------------
> >> mov eax,1
> >> mov ebx,0
> >> int 80
> >>
> >> On 29 Sep 2011, at 09:36, Alistair Young wrote:
> >>
> >>>> <transacted/> Should be after <from>
> >>> it is after from - do you mean it should be before?
> >>>  <route id="eDirSuccessBroadcast">
> >>>    <from uri="activemq:topic:edirectoryprocessed"/>
> >>>    <transacted />
> >>>    <process ref="groupwiseProcessor" />
> >>>    <to uri="activemq:topic:blackboard"/>
> >>>  </route>
> >>>
> >>> thanks for the dead letter tips, will apply them.
> >>>
> >>> Alistair
> >>>
> >>> --------------
> >>> mov eax,1
> >>> mov ebx,0
> >>> int 80
> >>>
> >>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
> >>>
> >>>> Hi
> >>>>
> >>>> <transacted/> Should be after <from>
> >>>>
> >>>>
> >>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
> >>>> <al...@uhi.ac.uk> wrote:
> >>>>>> Do you use message expiry?
> >>>>> no
> >>>>>
> >>>>>> timestamp plugin
> >>>>>
> >>>>> using that
> >>>>>
> >>>>> activemq 5.5.0
> >>>>> camel 2.8.0
> >>>>> spring 3.0.5
> >>>>>
> >>>>> noticed sl4j errors on startup, fixed that and now the tracer is
> logging so hopefully I can see any errors.
> >>>>>
> >>>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
> >>>>>   <from uri="activemq:topic:matrix"/>
> >>>>>   <process ref="matrixProcessor" />
> >>>>>   <transacted />
> >>>>>   <to uri="activemq:topic:edirectory"/>
> >>>>> </route>
> >>>>>
> >>>>>     <bean id="jmsConnectionFactory"
> class="org.apache.activemq.ActiveMQConnectionFactory"
> depends-on="matrixBrokerID">
> >>>>>             <property name="brokerURL"
> value="vm://matrixBroker?create=false"/>
> >>>>>     </bean>
> >>>>>
> >>>>>     <bean id="jmsTransactionManager"
> class="org.springframework.jms.connection.JmsTransactionManager">
> >>>>>             <property name="connectionFactory"
> ref="jmsConnectionFactory"/>
> >>>>>     </bean>
> >>>>>
> >>>>>     <bean id="activemq"
> class="org.apache.activemq.camel.component.ActiveMQComponent">
> >>>>>             <property name="connectionFactory"
> ref="jmsConnectionFactory"/>
> >>>>>             <property name="transacted" value="true"/>
> >>>>>             <property name="transactionManager"
> ref="jmsTransactionManager"/>
> >>>>>     </bean>
> >>>>>
> >>>>> <bean id="matrixDeadLetterErrorHandler"
> class="org.apache.camel.builder.DeadLetterChannelBuilder">
> >>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
> >>>>> <property name="redeliveryPolicy"
> ref="matrixRedeliveryPolicyConfig"/>
> >>>>> </bean>
> >>>>>
> >>>>> <bean id="matrixRedeliveryPolicyConfig"
> class="org.apache.camel.processor.RedeliveryPolicy">
> >>>>> <property name="maximumRedeliveries" value="10"/>
> >>>>> <property name="redeliveryDelay" value="250"/>
> >>>>> </bean>
> >>>>>
> >>>>> thanks,
> >>>>>
> >>>>> Alistair
> >>>>>
> >>>>>
> >>>>> --------------
> >>>>> mov eax,1
> >>>>> mov ebx,0
> >>>>> int 80
> >>>>>
> >>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
> >>>>>
> >>>>>> Hi
> >>>>>>
> >>>>>> Do you use message expiry?
> >>>>>> Make sure clocks between server/clients is synced as much as
> possible.
> >>>>>>
> >>>>>> There is a timestamp plugin
> >>>>>> http://activemq.apache.org/timestampplugin.html
> >>>>>>
> >>>>>> And do you use queue or topic.
> >>>>>> What version of AMQ and Camel are you using?
> >>>>>> And how have you configured the AMQ broker, and the Camel context?
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com>
> wrote:
> >>>>>>> Hi
> >>>>>>>
> >>>>>>> Where the logs go, if it's logged at all, still depends on your
> logger and
> >>>>>>> how you configured it.
> >>>>>>>
> >>>>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
> >>>>>>>
> >>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
> >>>>>>> [2]http://camel.apache.org/logging-questions.html
> >>>>>>>
> >>>>>>> Taariq
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
> alistair.young@uhi.ac.uk>wrote:
> >>>>>>>
> >>>>>>>> which is the best trace method to use? trace="true", or
> camelTracer and
> >>>>>>>> traceFormatter beans? and where does the log end up? I've tried
> them all but
> >>>>>>>> no log appears.
> >>>>>>>>
> >>>>>>>> Alistair
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> mov eax,1
> >>>>>>>> mov ebx,0
> >>>>>>>> int 80h
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
> >>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> I suggest enable tracing to see exactly what happens in your
> route.
> >>>>>>>>>
> >>>>>>>>> regards, Marco
> >>>>>>>>>
> >>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
> >>>>>>>>>> I now have a dead letter channel which is empty after losing 9
> out of 10
> >>>>>>>> messages. I also added a logging handler which logged nothing.
> Verified the
> >>>>>>>> messages arrived at the broker, then they just vanished.
> >>>>>>>>>>
> >>>>>>>>>> Alistair
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Claus Ibsen
> >>>>>> -----------------
> >>>>>> FuseSource
> >>>>>> Email: cibsen@fusesource.com
> >>>>>> Web: http://fusesource.com
> >>>>>> Twitter: davsclaus, fusenews
> >>>>>> Blog: http://davsclaus.blogspot.com/
> >>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Claus Ibsen
> >>>> -----------------
> >>>> FuseSource
> >>>> Email: cibsen@fusesource.com
> >>>> Web: http://fusesource.com
> >>>> Twitter: davsclaus, fusenews
> >>>> Blog: http://davsclaus.blogspot.com/
> >>>> Author of Camel in Action: http://www.manning.com/ibsen/
> >>>
> >>
> >
>
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
dunno - nothing works. Random messages are just vanishing once they reach the broker. No trace, no logs, no dead letter queue. Just vanishing. I've removed <transacted /> and <process> but it doesn't help. The producer is a few secs behind the broker:

sent : 11:25:26
arrived : 11:24:57
timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably the timestampplugin doing this
message vanishes

but all messages display this clock behaviour and not all vanish.

Alistair

--------------
mov eax,1
mov ebx,0
int 80

On 29 Sep 2011, at 10:24, Alistair Young wrote:

> just saw your info about transacted being before from - will change that and monitor again.
> 
> thanks,
> 
> Alistair
> 
> --------------
> mov eax,1
> mov ebx,0
> int 80
> 
> On 29 Sep 2011, at 10:18, Alistair Young wrote:
> 
>> just noticed a batch of identical 5 messages, three were missing and another single message vanished. tracer logged nothing. No errors, dead letter queue empty.
>> 
>> One thing that happens is another machine polls the stats topic in activemq every 2mins. Would that cause a problem? It asks for stats on the matrix topic, which is part of the transacted route.
>> 
>> 29 September 2011 10:05:07 - Adding destination : Topic:ActiveMQ.Advisory.Connection
>> 29 September 2011 10:05:07 - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
>> 29 September 2011 10:05:07 - Stopping connection: vm://matrixBroker#285916
>> 29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
>> 29 September 2011 10:05:07 - Connection Stopped: vm://matrixBroker#285916
>> 29 September 2011 10:05:07 - Setting up new connection id: ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address: vm://matrixBroker#285920
>> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo {commandId = 1, responseRequired = true, connectionId = ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId = ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null, password = *****, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true, faultTolerant = false}
>> 
>> Alistair
>> 
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>> 
>> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>> 
>>>> <transacted/> Should be after <from>
>>> it is after from - do you mean it should be before?
>>>  <route id="eDirSuccessBroadcast">
>>>    <from uri="activemq:topic:edirectoryprocessed"/>
>>>    <transacted />
>>>    <process ref="groupwiseProcessor" />
>>>    <to uri="activemq:topic:blackboard"/>
>>>  </route>
>>> 
>>> thanks for the dead letter tips, will apply them.
>>> 
>>> Alistair
>>> 
>>> --------------
>>> mov eax,1
>>> mov ebx,0
>>> int 80
>>> 
>>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>> 
>>>> Hi
>>>> 
>>>> <transacted/> Should be after <from>
>>>> 
>>>> 
>>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>>> <al...@uhi.ac.uk> wrote:
>>>>>> Do you use message expiry?
>>>>> no
>>>>> 
>>>>>> timestamp plugin
>>>>> 
>>>>> using that
>>>>> 
>>>>> activemq 5.5.0
>>>>> camel 2.8.0
>>>>> spring 3.0.5
>>>>> 
>>>>> noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.
>>>>> 
>>>>> <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>>   <from uri="activemq:topic:matrix"/>
>>>>>   <process ref="matrixProcessor" />
>>>>>   <transacted />
>>>>>   <to uri="activemq:topic:edirectory"/>
>>>>> </route>
>>>>> 
>>>>>     <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
>>>>>             <property name="brokerURL" value="vm://matrixBroker?create=false"/>
>>>>>     </bean>
>>>>> 
>>>>>     <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
>>>>>             <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>>>>     </bean>
>>>>> 
>>>>>     <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>>             <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>>>>             <property name="transacted" value="true"/>
>>>>>             <property name="transactionManager" ref="jmsTransactionManager"/>
>>>>>     </bean>
>>>>> 
>>>>> <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>> <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>> <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
>>>>> </bean>
>>>>> 
>>>>> <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
>>>>> <property name="maximumRedeliveries" value="10"/>
>>>>> <property name="redeliveryDelay" value="250"/>
>>>>> </bean>
>>>>> 
>>>>> thanks,
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> 
>>>>> --------------
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80
>>>>> 
>>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>> 
>>>>>> Hi
>>>>>> 
>>>>>> Do you use message expiry?
>>>>>> Make sure clocks between server/clients is synced as much as possible.
>>>>>> 
>>>>>> There is a timestamp plugin
>>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>> 
>>>>>> And do you use queue or topic.
>>>>>> What version of AMQ and Camel are you using?
>>>>>> And how have you configured the AMQ broker, and the Camel context?
>>>>>> 
>>>>>> 
>>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com> wrote:
>>>>>>> Hi
>>>>>>> 
>>>>>>> Where the logs go, if it's logged at all, still depends on your logger and
>>>>>>> how you configured it.
>>>>>>> 
>>>>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>>>>> 
>>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>> 
>>>>>>> Taariq
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>>>>>>> 
>>>>>>>> which is the best trace method to use? trace="true", or camelTracer and
>>>>>>>> traceFormatter beans? and where does the log end up? I've tried them all but
>>>>>>>> no log appears.
>>>>>>>> 
>>>>>>>> Alistair
>>>>>>>> 
>>>>>>>> --
>>>>>>>> mov eax,1
>>>>>>>> mov ebx,0
>>>>>>>> int 80h
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I suggest enable tracing to see exactly what happens in your route.
>>>>>>>>> 
>>>>>>>>> regards, Marco
>>>>>>>>> 
>>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>>> I now have a dead letter channel which is empty after losing 9 out of 10
>>>>>>>> messages. I also added a logging handler which logged nothing. Verified the
>>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>> 
>>>>>>>>>> Alistair
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> -----------------
>>>>>> FuseSource
>>>>>> Email: cibsen@fusesource.com
>>>>>> Web: http://fusesource.com
>>>>>> Twitter: davsclaus, fusenews
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Claus Ibsen
>>>> -----------------
>>>> FuseSource
>>>> Email: cibsen@fusesource.com
>>>> Web: http://fusesource.com
>>>> Twitter: davsclaus, fusenews
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> 
>> 
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
just saw your info about transacted being before from - will change that and monitor again.

thanks,

Alistair

--------------
mov eax,1
mov ebx,0
int 80

On 29 Sep 2011, at 10:18, Alistair Young wrote:

> just noticed a batch of identical 5 messages, three were missing and another single message vanished. tracer logged nothing. No errors, dead letter queue empty.
> 
> One thing that happens is another machine polls the stats topic in activemq every 2mins. Would that cause a problem? It asks for stats on the matrix topic, which is part of the transacted route.
> 
> 29 September 2011 10:05:07 - Adding destination : Topic:ActiveMQ.Advisory.Connection
> 29 September 2011 10:05:07 - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
> 29 September 2011 10:05:07 - Stopping connection: vm://matrixBroker#285916
> 29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
> 29 September 2011 10:05:07 - Connection Stopped: vm://matrixBroker#285916
> 29 September 2011 10:05:07 - Setting up new connection id: ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address: vm://matrixBroker#285920
> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo {commandId = 1, responseRequired = true, connectionId = ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId = ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null, password = *****, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true, faultTolerant = false}
> 
> Alistair
> 
> --------------
> mov eax,1
> mov ebx,0
> int 80
> 
> On 29 Sep 2011, at 09:36, Alistair Young wrote:
> 
>>> <transacted/> Should be after <from>
>> it is after from - do you mean it should be before?
>>   <route id="eDirSuccessBroadcast">
>>     <from uri="activemq:topic:edirectoryprocessed"/>
>>     <transacted />
>>     <process ref="groupwiseProcessor" />
>>     <to uri="activemq:topic:blackboard"/>
>>   </route>
>> 
>> thanks for the dead letter tips, will apply them.
>> 
>> Alistair
>> 
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>> 
>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>> 
>>> Hi
>>> 
>>> <transacted/> Should be after <from>
>>> 
>>> 
>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>> <al...@uhi.ac.uk> wrote:
>>>>> Do you use message expiry?
>>>> no
>>>> 
>>>>> timestamp plugin
>>>> 
>>>> using that
>>>> 
>>>> activemq 5.5.0
>>>> camel 2.8.0
>>>> spring 3.0.5
>>>> 
>>>> noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.
>>>> 
>>>>  <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>    <from uri="activemq:topic:matrix"/>
>>>>    <process ref="matrixProcessor" />
>>>>    <transacted />
>>>>    <to uri="activemq:topic:edirectory"/>
>>>>  </route>
>>>> 
>>>>      <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
>>>>              <property name="brokerURL" value="vm://matrixBroker?create=false"/>
>>>>      </bean>
>>>> 
>>>>      <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
>>>>              <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>>>      </bean>
>>>> 
>>>>      <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>              <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>>>              <property name="transacted" value="true"/>
>>>>              <property name="transactionManager" ref="jmsTransactionManager"/>
>>>>      </bean>
>>>> 
>>>> <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>  <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>  <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
>>>> </bean>
>>>> 
>>>> <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
>>>>  <property name="maximumRedeliveries" value="10"/>
>>>>  <property name="redeliveryDelay" value="250"/>
>>>> </bean>
>>>> 
>>>> thanks,
>>>> 
>>>> Alistair
>>>> 
>>>> 
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>> 
>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>> 
>>>>> Hi
>>>>> 
>>>>> Do you use message expiry?
>>>>> Make sure clocks between server/clients is synced as much as possible.
>>>>> 
>>>>> There is a timestamp plugin
>>>>> http://activemq.apache.org/timestampplugin.html
>>>>> 
>>>>> And do you use queue or topic.
>>>>> What version of AMQ and Camel are you using?
>>>>> And how have you configured the AMQ broker, and the Camel context?
>>>>> 
>>>>> 
>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com> wrote:
>>>>>> Hi
>>>>>> 
>>>>>> Where the logs go, if it's logged at all, still depends on your logger and
>>>>>> how you configured it.
>>>>>> 
>>>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>>>> 
>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>> 
>>>>>> Taariq
>>>>>> 
>>>>>> 
>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>>>>>> 
>>>>>>> which is the best trace method to use? trace="true", or camelTracer and
>>>>>>> traceFormatter beans? and where does the log end up? I've tried them all but
>>>>>>> no log appears.
>>>>>>> 
>>>>>>> Alistair
>>>>>>> 
>>>>>>> --
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80h
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>> 
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> I suggest enable tracing to see exactly what happens in your route.
>>>>>>>> 
>>>>>>>> regards, Marco
>>>>>>>> 
>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>> I now have a dead letter channel which is empty after losing 9 out of 10
>>>>>>> messages. I also added a logging handler which logged nothing. Verified the
>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>> 
>>>>>>>>> Alistair
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Claus Ibsen
>>>>> -----------------
>>>>> FuseSource
>>>>> Email: cibsen@fusesource.com
>>>>> Web: http://fusesource.com
>>>>> Twitter: davsclaus, fusenews
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> Claus Ibsen
>>> -----------------
>>> FuseSource
>>> Email: cibsen@fusesource.com
>>> Web: http://fusesource.com
>>> Twitter: davsclaus, fusenews
>>> Blog: http://davsclaus.blogspot.com/
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>> 
>

Re: Messages being lost from route

Posted by Claus Ibsen <cl...@gmail.com>.
Hi

For topics you ought to set an unique clientId and subscription name
the consumer side.
http://camel.apache.org/jms
http://activemq.apache.org/how-do-durable-queues-and-topics-work.html



On Thu, Sep 29, 2011 at 11:18 AM, Alistair Young
<al...@uhi.ac.uk> wrote:
> just noticed a batch of identical 5 messages, three were missing and another single message vanished. tracer logged nothing. No errors, dead letter queue empty.
>
> One thing that happens is another machine polls the stats topic in activemq every 2mins. Would that cause a problem? It asks for stats on the matrix topic, which is part of the transacted route.
>
> 29 September 2011 10:05:07 - Adding destination : Topic:ActiveMQ.Advisory.Connection
> 29 September 2011 10:05:07 - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
> 29 September 2011 10:05:07 - Stopping connection: vm://matrixBroker#285916
> 29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
> 29 September 2011 10:05:07 - Connection Stopped: vm://matrixBroker#285916
> 29 September 2011 10:05:07 - Setting up new connection id: ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address: vm://matrixBroker#285920
> 29 September 2011 10:05:07 - Adding Connection : ConnectionInfo {commandId = 1, responseRequired = true, connectionId = ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId = ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null, password = *****, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true, faultTolerant = false}
>
> Alistair
>
> --------------
> mov eax,1
> mov ebx,0
> int 80
>
> On 29 Sep 2011, at 09:36, Alistair Young wrote:
>
>>> <transacted/> Should be after <from>
>> it is after from - do you mean it should be before?
>>    <route id="eDirSuccessBroadcast">
>>      <from uri="activemq:topic:edirectoryprocessed"/>
>>      <transacted />
>>      <process ref="groupwiseProcessor" />
>>      <to uri="activemq:topic:blackboard"/>
>>    </route>
>>
>> thanks for the dead letter tips, will apply them.
>>
>> Alistair
>>
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>>
>> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
>>
>>> Hi
>>>
>>> <transacted/> Should be after <from>
>>>
>>>
>>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>>> <al...@uhi.ac.uk> wrote:
>>>>> Do you use message expiry?
>>>> no
>>>>
>>>>> timestamp plugin
>>>>
>>>> using that
>>>>
>>>> activemq 5.5.0
>>>> camel 2.8.0
>>>> spring 3.0.5
>>>>
>>>> noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.
>>>>
>>>>   <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>>     <from uri="activemq:topic:matrix"/>
>>>>     <process ref="matrixProcessor" />
>>>>     <transacted />
>>>>     <to uri="activemq:topic:edirectory"/>
>>>>   </route>
>>>>
>>>>       <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
>>>>               <property name="brokerURL" value="vm://matrixBroker?create=false"/>
>>>>       </bean>
>>>>
>>>>       <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
>>>>               <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>>>       </bean>
>>>>
>>>>       <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>>               <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>>>               <property name="transacted" value="true"/>
>>>>               <property name="transactionManager" ref="jmsTransactionManager"/>
>>>>       </bean>
>>>>
>>>> <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>>   <property name="deadLetterUri" value="jms:queue:dead"/>
>>>>   <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
>>>> </bean>
>>>>
>>>> <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
>>>>   <property name="maximumRedeliveries" value="10"/>
>>>>   <property name="redeliveryDelay" value="250"/>
>>>> </bean>
>>>>
>>>> thanks,
>>>>
>>>> Alistair
>>>>
>>>>
>>>> --------------
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80
>>>>
>>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> Do you use message expiry?
>>>>> Make sure clocks between server/clients is synced as much as possible.
>>>>>
>>>>> There is a timestamp plugin
>>>>> http://activemq.apache.org/timestampplugin.html
>>>>>
>>>>> And do you use queue or topic.
>>>>> What version of AMQ and Camel are you using?
>>>>> And how have you configured the AMQ broker, and the Camel context?
>>>>>
>>>>>
>>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com> wrote:
>>>>>> Hi
>>>>>>
>>>>>> Where the logs go, if it's logged at all, still depends on your logger and
>>>>>> how you configured it.
>>>>>>
>>>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>>>>
>>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>>>
>>>>>> Taariq
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>>>>>>
>>>>>>> which is the best trace method to use? trace="true", or camelTracer and
>>>>>>> traceFormatter beans? and where does the log end up? I've tried them all but
>>>>>>> no log appears.
>>>>>>>
>>>>>>> Alistair
>>>>>>>
>>>>>>> --
>>>>>>> mov eax,1
>>>>>>> mov ebx,0
>>>>>>> int 80h
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I suggest enable tracing to see exactly what happens in your route.
>>>>>>>>
>>>>>>>> regards, Marco
>>>>>>>>
>>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>>> I now have a dead letter channel which is empty after losing 9 out of 10
>>>>>>> messages. I also added a logging handler which logged nothing. Verified the
>>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>>>
>>>>>>>>> Alistair
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> -----------------
>>>>> FuseSource
>>>>> Email: cibsen@fusesource.com
>>>>> Web: http://fusesource.com
>>>>> Twitter: davsclaus, fusenews
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> FuseSource
>>> Email: cibsen@fusesource.com
>>> Web: http://fusesource.com
>>> Twitter: davsclaus, fusenews
>>> Blog: http://davsclaus.blogspot.com/
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>
>
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
just noticed a batch of identical 5 messages, three were missing and another single message vanished. tracer logged nothing. No errors, dead letter queue empty.

One thing that happens is another machine polls the stats topic in activemq every 2mins. Would that cause a problem? It asks for stats on the matrix topic, which is part of the transacted route.

29 September 2011 10:05:07 - Adding destination : Topic:ActiveMQ.Advisory.Connection
29 September 2011 10:05:07 - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
29 September 2011 10:05:07 - Stopping connection: vm://matrixBroker#285916
29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
29 September 2011 10:05:07 - Connection Stopped: vm://matrixBroker#285916
29 September 2011 10:05:07 - Setting up new connection id: ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address: vm://matrixBroker#285920
29 September 2011 10:05:07 - Adding Connection : ConnectionInfo {commandId = 1, responseRequired = true, connectionId = ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId = ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null, password = *****, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true, faultTolerant = false}

Alistair

--------------
mov eax,1
mov ebx,0
int 80

On 29 Sep 2011, at 09:36, Alistair Young wrote:

>> <transacted/> Should be after <from>
> it is after from - do you mean it should be before?
>    <route id="eDirSuccessBroadcast">
>      <from uri="activemq:topic:edirectoryprocessed"/>
>      <transacted />
>      <process ref="groupwiseProcessor" />
>      <to uri="activemq:topic:blackboard"/>
>    </route>
> 
> thanks for the dead letter tips, will apply them.
> 
> Alistair
> 
> --------------
> mov eax,1
> mov ebx,0
> int 80
> 
> On 29 Sep 2011, at 09:20, Claus Ibsen wrote:
> 
>> Hi
>> 
>> <transacted/> Should be after <from>
>> 
>> 
>> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
>> <al...@uhi.ac.uk> wrote:
>>>> Do you use message expiry?
>>> no
>>> 
>>>> timestamp plugin
>>> 
>>> using that
>>> 
>>> activemq 5.5.0
>>> camel 2.8.0
>>> spring 3.0.5
>>> 
>>> noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.
>>> 
>>>   <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>>     <from uri="activemq:topic:matrix"/>
>>>     <process ref="matrixProcessor" />
>>>     <transacted />
>>>     <to uri="activemq:topic:edirectory"/>
>>>   </route>
>>> 
>>>       <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
>>>               <property name="brokerURL" value="vm://matrixBroker?create=false"/>
>>>       </bean>
>>> 
>>>       <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
>>>               <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>>       </bean>
>>> 
>>>       <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
>>>               <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>>               <property name="transacted" value="true"/>
>>>               <property name="transactionManager" ref="jmsTransactionManager"/>
>>>       </bean>
>>> 
>>> <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>>   <property name="deadLetterUri" value="jms:queue:dead"/>
>>>   <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
>>> </bean>
>>> 
>>> <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
>>>   <property name="maximumRedeliveries" value="10"/>
>>>   <property name="redeliveryDelay" value="250"/>
>>> </bean>
>>> 
>>> thanks,
>>> 
>>> Alistair
>>> 
>>> 
>>> --------------
>>> mov eax,1
>>> mov ebx,0
>>> int 80
>>> 
>>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>>> 
>>>> Hi
>>>> 
>>>> Do you use message expiry?
>>>> Make sure clocks between server/clients is synced as much as possible.
>>>> 
>>>> There is a timestamp plugin
>>>> http://activemq.apache.org/timestampplugin.html
>>>> 
>>>> And do you use queue or topic.
>>>> What version of AMQ and Camel are you using?
>>>> And how have you configured the AMQ broker, and the Camel context?
>>>> 
>>>> 
>>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com> wrote:
>>>>> Hi
>>>>> 
>>>>> Where the logs go, if it's logged at all, still depends on your logger and
>>>>> how you configured it.
>>>>> 
>>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>>> 
>>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>>> [2]http://camel.apache.org/logging-questions.html
>>>>> 
>>>>> Taariq
>>>>> 
>>>>> 
>>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>>>>> 
>>>>>> which is the best trace method to use? trace="true", or camelTracer and
>>>>>> traceFormatter beans? and where does the log end up? I've tried them all but
>>>>>> no log appears.
>>>>>> 
>>>>>> Alistair
>>>>>> 
>>>>>> --
>>>>>> mov eax,1
>>>>>> mov ebx,0
>>>>>> int 80h
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I suggest enable tracing to see exactly what happens in your route.
>>>>>>> 
>>>>>>> regards, Marco
>>>>>>> 
>>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>>> I now have a dead letter channel which is empty after losing 9 out of 10
>>>>>> messages. I also added a logging handler which logged nothing. Verified the
>>>>>> messages arrived at the broker, then they just vanished.
>>>>>>>> 
>>>>>>>> Alistair
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> FuseSource
>>>> Email: cibsen@fusesource.com
>>>> Web: http://fusesource.com
>>>> Twitter: davsclaus, fusenews
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> 
>>> 
>> 
>> 
>> 
>> -- 
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
> <transacted/> Should be after <from>
it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
      <from uri="activemq:topic:edirectoryprocessed"/>
      <transacted />
      <process ref="groupwiseProcessor" />
      <to uri="activemq:topic:blackboard"/>
    </route>

thanks for the dead letter tips, will apply them.

Alistair

--------------
mov eax,1
mov ebx,0
int 80

On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

> Hi
> 
> <transacted/> Should be after <from>
> 
> 
> On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
> <al...@uhi.ac.uk> wrote:
>>> Do you use message expiry?
>> no
>> 
>>> timestamp plugin
>> 
>> using that
>> 
>> activemq 5.5.0
>> camel 2.8.0
>> spring 3.0.5
>> 
>> noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.
>> 
>>    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>>      <from uri="activemq:topic:matrix"/>
>>      <process ref="matrixProcessor" />
>>      <transacted />
>>      <to uri="activemq:topic:edirectory"/>
>>    </route>
>> 
>>        <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
>>                <property name="brokerURL" value="vm://matrixBroker?create=false"/>
>>        </bean>
>> 
>>        <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
>>                <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>        </bean>
>> 
>>        <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
>>                <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>                <property name="transacted" value="true"/>
>>                <property name="transactionManager" ref="jmsTransactionManager"/>
>>        </bean>
>> 
>>  <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
>>    <property name="deadLetterUri" value="jms:queue:dead"/>
>>    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
>>  </bean>
>> 
>>  <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
>>    <property name="maximumRedeliveries" value="10"/>
>>    <property name="redeliveryDelay" value="250"/>
>>  </bean>
>> 
>> thanks,
>> 
>> Alistair
>> 
>> 
>> --------------
>> mov eax,1
>> mov ebx,0
>> int 80
>> 
>> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>> 
>>> Hi
>>> 
>>> Do you use message expiry?
>>> Make sure clocks between server/clients is synced as much as possible.
>>> 
>>> There is a timestamp plugin
>>> http://activemq.apache.org/timestampplugin.html
>>> 
>>> And do you use queue or topic.
>>> What version of AMQ and Camel are you using?
>>> And how have you configured the AMQ broker, and the Camel context?
>>> 
>>> 
>>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com> wrote:
>>>> Hi
>>>> 
>>>> Where the logs go, if it's logged at all, still depends on your logger and
>>>> how you configured it.
>>>> 
>>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>> 
>>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>>> [2]http://camel.apache.org/logging-questions.html
>>>> 
>>>> Taariq
>>>> 
>>>> 
>>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>>>> 
>>>>> which is the best trace method to use? trace="true", or camelTracer and
>>>>> traceFormatter beans? and where does the log end up? I've tried them all but
>>>>> no log appears.
>>>>> 
>>>>> Alistair
>>>>> 
>>>>> --
>>>>> mov eax,1
>>>>> mov ebx,0
>>>>> int 80h
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I suggest enable tracing to see exactly what happens in your route.
>>>>>> 
>>>>>> regards, Marco
>>>>>> 
>>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>>> I now have a dead letter channel which is empty after losing 9 out of 10
>>>>> messages. I also added a logging handler which logged nothing. Verified the
>>>>> messages arrived at the broker, then they just vanished.
>>>>>>> 
>>>>>>> Alistair
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Claus Ibsen
>>> -----------------
>>> FuseSource
>>> Email: cibsen@fusesource.com
>>> Web: http://fusesource.com
>>> Twitter: davsclaus, fusenews
>>> Blog: http://davsclaus.blogspot.com/
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>> 
>> 
> 
> 
> 
> -- 
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/

Re: Messages being lost from route

Posted by Claus Ibsen <cl...@gmail.com>.
Hi

<transacted/> Should be after <from>


On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
<al...@uhi.ac.uk> wrote:
>> Do you use message expiry?
> no
>
>> timestamp plugin
>
> using that
>
> activemq 5.5.0
> camel 2.8.0
> spring 3.0.5
>
> noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.
>
>    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>      <from uri="activemq:topic:matrix"/>
>      <process ref="matrixProcessor" />
>      <transacted />
>      <to uri="activemq:topic:edirectory"/>
>    </route>
>
>        <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
>                <property name="brokerURL" value="vm://matrixBroker?create=false"/>
>        </bean>
>
>        <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
>                <property name="connectionFactory" ref="jmsConnectionFactory"/>
>        </bean>
>
>        <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
>                <property name="connectionFactory" ref="jmsConnectionFactory"/>
>                <property name="transacted" value="true"/>
>                <property name="transactionManager" ref="jmsTransactionManager"/>
>        </bean>
>
>  <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
>    <property name="deadLetterUri" value="jms:queue:dead"/>
>    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
>  </bean>
>
>  <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
>    <property name="maximumRedeliveries" value="10"/>
>    <property name="redeliveryDelay" value="250"/>
>  </bean>
>
> thanks,
>
> Alistair
>
>
> --------------
> mov eax,1
> mov ebx,0
> int 80
>
> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>
>> Hi
>>
>> Do you use message expiry?
>> Make sure clocks between server/clients is synced as much as possible.
>>
>> There is a timestamp plugin
>> http://activemq.apache.org/timestampplugin.html
>>
>> And do you use queue or topic.
>> What version of AMQ and Camel are you using?
>> And how have you configured the AMQ broker, and the Camel context?
>>
>>
>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com> wrote:
>>> Hi
>>>
>>> Where the logs go, if it's logged at all, still depends on your logger and
>>> how you configured it.
>>>
>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>
>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>> [2]http://camel.apache.org/logging-questions.html
>>>
>>> Taariq
>>>
>>>
>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>>>
>>>> which is the best trace method to use? trace="true", or camelTracer and
>>>> traceFormatter beans? and where does the log end up? I've tried them all but
>>>> no log appears.
>>>>
>>>> Alistair
>>>>
>>>> --
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80h
>>>>
>>>>
>>>>
>>>>
>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I suggest enable tracing to see exactly what happens in your route.
>>>>>
>>>>> regards, Marco
>>>>>
>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>> I now have a dead letter channel which is empty after losing 9 out of 10
>>>> messages. I also added a logging handler which logged nothing. Verified the
>>>> messages arrived at the broker, then they just vanished.
>>>>>>
>>>>>> Alistair
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: Messages being lost from route

Posted by Claus Ibsen <cl...@gmail.com>.
And its better to use the native redelivery / dead letter queue,
features of the AMQ message broker, than from Camel.
http://activemq.apache.org/message-redelivery-and-dlq-handling.html
http://activemq.apache.org/redelivery-policy.html

So remove the matrixDeadLetterErrorHandler and use the DLQ from the AMQ instead.


On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
<al...@uhi.ac.uk> wrote:
>> Do you use message expiry?
> no
>
>> timestamp plugin
>
> using that
>
> activemq 5.5.0
> camel 2.8.0
> spring 3.0.5
>
> noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.
>
>    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
>      <from uri="activemq:topic:matrix"/>
>      <process ref="matrixProcessor" />
>      <transacted />
>      <to uri="activemq:topic:edirectory"/>
>    </route>
>
>        <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
>                <property name="brokerURL" value="vm://matrixBroker?create=false"/>
>        </bean>
>
>        <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
>                <property name="connectionFactory" ref="jmsConnectionFactory"/>
>        </bean>
>
>        <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
>                <property name="connectionFactory" ref="jmsConnectionFactory"/>
>                <property name="transacted" value="true"/>
>                <property name="transactionManager" ref="jmsTransactionManager"/>
>        </bean>
>
>  <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
>    <property name="deadLetterUri" value="jms:queue:dead"/>
>    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
>  </bean>
>
>  <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
>    <property name="maximumRedeliveries" value="10"/>
>    <property name="redeliveryDelay" value="250"/>
>  </bean>
>
> thanks,
>
> Alistair
>
>
> --------------
> mov eax,1
> mov ebx,0
> int 80
>
> On 29 Sep 2011, at 08:53, Claus Ibsen wrote:
>
>> Hi
>>
>> Do you use message expiry?
>> Make sure clocks between server/clients is synced as much as possible.
>>
>> There is a timestamp plugin
>> http://activemq.apache.org/timestampplugin.html
>>
>> And do you use queue or topic.
>> What version of AMQ and Camel are you using?
>> And how have you configured the AMQ broker, and the Camel context?
>>
>>
>> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com> wrote:
>>> Hi
>>>
>>> Where the logs go, if it's logged at all, still depends on your logger and
>>> how you configured it.
>>>
>>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>>>
>>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>>> [2]http://camel.apache.org/logging-questions.html
>>>
>>> Taariq
>>>
>>>
>>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>>>
>>>> which is the best trace method to use? trace="true", or camelTracer and
>>>> traceFormatter beans? and where does the log end up? I've tried them all but
>>>> no log appears.
>>>>
>>>> Alistair
>>>>
>>>> --
>>>> mov eax,1
>>>> mov ebx,0
>>>> int 80h
>>>>
>>>>
>>>>
>>>>
>>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I suggest enable tracing to see exactly what happens in your route.
>>>>>
>>>>> regards, Marco
>>>>>
>>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>>> I now have a dead letter channel which is empty after losing 9 out of 10
>>>> messages. I also added a logging handler which logged nothing. Verified the
>>>> messages arrived at the broker, then they just vanished.
>>>>>>
>>>>>> Alistair
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
> Do you use message expiry?
no

> timestamp plugin

using that

activemq 5.5.0
camel 2.8.0
spring 3.0.5

noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
      <from uri="activemq:topic:matrix"/>
      <process ref="matrixProcessor" />
      <transacted />
      <to uri="activemq:topic:edirectory"/>
    </route>

        <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
                <property name="brokerURL" value="vm://matrixBroker?create=false"/>
        </bean>

        <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
                <property name="connectionFactory" ref="jmsConnectionFactory"/>
        </bean>

        <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
                <property name="connectionFactory" ref="jmsConnectionFactory"/>
                <property name="transacted" value="true"/>
                <property name="transactionManager" ref="jmsTransactionManager"/>
        </bean>

  <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
  </bean>

  <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
  </bean>

thanks,

Alistair


--------------
mov eax,1
mov ebx,0
int 80

On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

> Hi
> 
> Do you use message expiry?
> Make sure clocks between server/clients is synced as much as possible.
> 
> There is a timestamp plugin
> http://activemq.apache.org/timestampplugin.html
> 
> And do you use queue or topic.
> What version of AMQ and Camel are you using?
> And how have you configured the AMQ broker, and the Camel context?
> 
> 
> On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com> wrote:
>> Hi
>> 
>> Where the logs go, if it's logged at all, still depends on your logger and
>> how you configured it.
>> 
>> Here are links to how to enable logging[1] and camel logging FAQ[2]
>> 
>> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
>> [2]http://camel.apache.org/logging-questions.html
>> 
>> Taariq
>> 
>> 
>> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>> 
>>> which is the best trace method to use? trace="true", or camelTracer and
>>> traceFormatter beans? and where does the log end up? I've tried them all but
>>> no log appears.
>>> 
>>> Alistair
>>> 
>>> --
>>> mov eax,1
>>> mov ebx,0
>>> int 80h
>>> 
>>> 
>>> 
>>> 
>>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I suggest enable tracing to see exactly what happens in your route.
>>>> 
>>>> regards, Marco
>>>> 
>>>> Am 28.09.2011 13:01, schrieb Alistair Young:
>>>>> I now have a dead letter channel which is empty after losing 9 out of 10
>>> messages. I also added a logging handler which logged nothing. Verified the
>>> messages arrived at the broker, then they just vanished.
>>>>> 
>>>>> Alistair
>>>>> 
>>>> 
>>> 
>>> 
>> 
> 
> 
> 
> -- 
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/

Re: Messages being lost from route

Posted by Claus Ibsen <cl...@gmail.com>.
Hi

Do you use message expiry?
Make sure clocks between server/clients is synced as much as possible.

There is a timestamp plugin
http://activemq.apache.org/timestampplugin.html

And do you use queue or topic.
What version of AMQ and Camel are you using?
And how have you configured the AMQ broker, and the Camel context?


On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <ta...@gmail.com> wrote:
> Hi
>
> Where the logs go, if it's logged at all, still depends on your logger and
> how you configured it.
>
> Here are links to how to enable logging[1] and camel logging FAQ[2]
>
> [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
> [2]http://camel.apache.org/logging-questions.html
>
> Taariq
>
>
> On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:
>
>> which is the best trace method to use? trace="true", or camelTracer and
>> traceFormatter beans? and where does the log end up? I've tried them all but
>> no log appears.
>>
>> Alistair
>>
>> --
>> mov eax,1
>> mov ebx,0
>> int 80h
>>
>>
>>
>>
>> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>>
>> > Hi,
>> >
>> > I suggest enable tracing to see exactly what happens in your route.
>> >
>> > regards, Marco
>> >
>> > Am 28.09.2011 13:01, schrieb Alistair Young:
>> >> I now have a dead letter channel which is empty after losing 9 out of 10
>> messages. I also added a logging handler which logged nothing. Verified the
>> messages arrived at the broker, then they just vanished.
>> >>
>> >> Alistair
>> >>
>> >
>>
>>
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: Messages being lost from route

Posted by Taariq Levack <ta...@gmail.com>.
Hi

Where the logs go, if it's logged at all, still depends on your logger and
how you configured it.

Here are links to how to enable logging[1] and camel logging FAQ[2]

[1]http://camel.apache.org/how-do-i-enable-debug-logging.html
[2]http://camel.apache.org/logging-questions.html

Taariq


On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <al...@uhi.ac.uk>wrote:

> which is the best trace method to use? trace="true", or camelTracer and
> traceFormatter beans? and where does the log end up? I've tried them all but
> no log appears.
>
> Alistair
>
> --
> mov eax,1
> mov ebx,0
> int 80h
>
>
>
>
> On 28 Sep 2011, at 12:08, Marco Westermann wrote:
>
> > Hi,
> >
> > I suggest enable tracing to see exactly what happens in your route.
> >
> > regards, Marco
> >
> > Am 28.09.2011 13:01, schrieb Alistair Young:
> >> I now have a dead letter channel which is empty after losing 9 out of 10
> messages. I also added a logging handler which logged nothing. Verified the
> messages arrived at the broker, then they just vanished.
> >>
> >> Alistair
> >>
> >
>
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
which is the best trace method to use? trace="true", or camelTracer and traceFormatter beans? and where does the log end up? I've tried them all but no log appears.

Alistair

-- 
mov eax,1
mov ebx,0
int 80h




On 28 Sep 2011, at 12:08, Marco Westermann wrote:

> Hi,
> 
> I suggest enable tracing to see exactly what happens in your route.
> 
> regards, Marco
> 
> Am 28.09.2011 13:01, schrieb Alistair Young:
>> I now have a dead letter channel which is empty after losing 9 out of 10 messages. I also added a logging handler which logged nothing. Verified the messages arrived at the broker, then they just vanished.
>> 
>> Alistair
>> 
>

Re: Messages being lost from route

Posted by Marco Westermann <Ma...@gmx.de>.
Hi,

I suggest enable tracing to see exactly what happens in your route.

regards, Marco

Am 28.09.2011 13:01, schrieb Alistair Young:
> I now have a dead letter channel which is empty after losing 9 out of 10 messages. I also added a logging handler which logged nothing. Verified the messages arrived at the broker, then they just vanished.
>
> Alistair
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
I now have a dead letter channel which is empty after losing 9 out of 10 messages. I also added a logging handler which logged nothing. Verified the messages arrived at the broker, then they just vanished.

Alistair

-- 
mov eax,1
mov ebx,0
int 80h




On 28 Sep 2011, at 11:42, Alistair Young wrote:

> just noticed this:
> 
> Default Error Handler:
> "whenever an Exchange could not be processed the error is propagated back to the client"
> 
> the client sends to topic://A and camel routes to topic://B. If the routing fails will the Default Error Handler tell the client, even though the client doesn't know about topic://B ?
> 
> the client does not receive any errors for the disappearing messages.
> 
> Alistair
> 
> -- 
> mov eax,1
> mov ebx,0
> int 80h
> 
> 
> 
> 
> On 28 Sep 2011, at 11:21, Mick Knutson wrote:
> 
>> Do you have a deadLetterChannel setup in case of message exceptions?
>> 
>> ---
>> Thank You…
>> 
>> Mick Knutson, President
>> 
>> BASE Logic, Inc.
>> Enterprise Architecture, Design, Mentoring & Agile Consulting
>> p. (855) BASE-LOGIC: (227-3564-42)
>> p. (478) BASE-LOGIC (227-3564-42)
>> f. (855) BASE-LOGIC: (227-3564-42)
>> 
>> Website: http://www.baselogic.com
>> Blog: http://www.baselogic.com/blog/
>> Linked IN: http://linkedin.com/in/mickknutson
>> Twitter: http://twitter.com/mickknutson
>> ---
>> 
>> 
>> 
>> On Wed, Sep 28, 2011 at 6:16 AM, Alistair Young <Al...@uhi.ac.uk>wrote:
>> 
>>> Hi there,
>>> 
>>> is there anything that would cause camel to ignore messages in a route? I'm
>>> getting persistent messages simply disappearing from a transacted route.
>>> I've verified the messages arrive at the activemq broker (tcpdump) and also
>>> that on one occasion, 5 out of 6 of the messages just disappeared. They
>>> didn't make it into the route as the processor didn't see them and the
>>> consumer never saw them either. They just vanished. Restarting the broker
>>> seems to ease the situation until the next disappearing messages and it gets
>>> worse and worse.
>>> 
>>> thanks,
>>> 
>>> Alistair
>>> 
>>> --
>>> mov eax,1
>>> mov ebx,0
>>> int 80h
>>> 
>>> 
>>> 
>>> 
>>> 
>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
just noticed this:

Default Error Handler:
"whenever an Exchange could not be processed the error is propagated back to the client"

the client sends to topic://A and camel routes to topic://B. If the routing fails will the Default Error Handler tell the client, even though the client doesn't know about topic://B ?

the client does not receive any errors for the disappearing messages.

Alistair

-- 
mov eax,1
mov ebx,0
int 80h




On 28 Sep 2011, at 11:21, Mick Knutson wrote:

> Do you have a deadLetterChannel setup in case of message exceptions?
> 
> ---
> Thank You…
> 
> Mick Knutson, President
> 
> BASE Logic, Inc.
> Enterprise Architecture, Design, Mentoring & Agile Consulting
> p. (855) BASE-LOGIC: (227-3564-42)
> p. (478) BASE-LOGIC (227-3564-42)
> f. (855) BASE-LOGIC: (227-3564-42)
> 
> Website: http://www.baselogic.com
> Blog: http://www.baselogic.com/blog/
> Linked IN: http://linkedin.com/in/mickknutson
> Twitter: http://twitter.com/mickknutson
> ---
> 
> 
> 
> On Wed, Sep 28, 2011 at 6:16 AM, Alistair Young <Al...@uhi.ac.uk>wrote:
> 
>> Hi there,
>> 
>> is there anything that would cause camel to ignore messages in a route? I'm
>> getting persistent messages simply disappearing from a transacted route.
>> I've verified the messages arrive at the activemq broker (tcpdump) and also
>> that on one occasion, 5 out of 6 of the messages just disappeared. They
>> didn't make it into the route as the processor didn't see them and the
>> consumer never saw them either. They just vanished. Restarting the broker
>> seems to ease the situation until the next disappearing messages and it gets
>> worse and worse.
>> 
>> thanks,
>> 
>> Alistair
>> 
>> --
>> mov eax,1
>> mov ebx,0
>> int 80h
>> 
>> 
>> 
>> 
>>

Re: Messages being lost from route

Posted by Alistair Young <al...@uhi.ac.uk>.
thanks for that. Have set one up now and will watch it.

Alistair

-- 
mov eax,1
mov ebx,0
int 80h




On 28 Sep 2011, at 11:21, Mick Knutson wrote:

> Do you have a deadLetterChannel setup in case of message exceptions?
> 
> ---
> Thank You…
> 
> Mick Knutson, President
> 
> BASE Logic, Inc.
> Enterprise Architecture, Design, Mentoring & Agile Consulting
> p. (855) BASE-LOGIC: (227-3564-42)
> p. (478) BASE-LOGIC (227-3564-42)
> f. (855) BASE-LOGIC: (227-3564-42)
> 
> Website: http://www.baselogic.com
> Blog: http://www.baselogic.com/blog/
> Linked IN: http://linkedin.com/in/mickknutson
> Twitter: http://twitter.com/mickknutson
> ---
> 
> 
> 
> On Wed, Sep 28, 2011 at 6:16 AM, Alistair Young <Al...@uhi.ac.uk>wrote:
> 
>> Hi there,
>> 
>> is there anything that would cause camel to ignore messages in a route? I'm
>> getting persistent messages simply disappearing from a transacted route.
>> I've verified the messages arrive at the activemq broker (tcpdump) and also
>> that on one occasion, 5 out of 6 of the messages just disappeared. They
>> didn't make it into the route as the processor didn't see them and the
>> consumer never saw them either. They just vanished. Restarting the broker
>> seems to ease the situation until the next disappearing messages and it gets
>> worse and worse.
>> 
>> thanks,
>> 
>> Alistair
>> 
>> --
>> mov eax,1
>> mov ebx,0
>> int 80h
>> 
>> 
>> 
>> 
>>

Re: Messages being lost from route

Posted by Mick Knutson <mk...@baselogic.com>.
Do you have a deadLetterChannel setup in case of message exceptions?

---
Thank You…

Mick Knutson, President

BASE Logic, Inc.
Enterprise Architecture, Design, Mentoring & Agile Consulting
p. (855) BASE-LOGIC: (227-3564-42)
p. (478) BASE-LOGIC (227-3564-42)
f. (855) BASE-LOGIC: (227-3564-42)

Website: http://www.baselogic.com
Blog: http://www.baselogic.com/blog/
Linked IN: http://linkedin.com/in/mickknutson
Twitter: http://twitter.com/mickknutson
---



On Wed, Sep 28, 2011 at 6:16 AM, Alistair Young <Al...@uhi.ac.uk>wrote:

> Hi there,
>
> is there anything that would cause camel to ignore messages in a route? I'm
> getting persistent messages simply disappearing from a transacted route.
> I've verified the messages arrive at the activemq broker (tcpdump) and also
> that on one occasion, 5 out of 6 of the messages just disappeared. They
> didn't make it into the route as the processor didn't see them and the
> consumer never saw them either. They just vanished. Restarting the broker
> seems to ease the situation until the next disappearing messages and it gets
> worse and worse.
>
> thanks,
>
> Alistair
>
> --
> mov eax,1
> mov ebx,0
> int 80h
>
>
>
>
>