Re: Best way to build a wait/retry loop with Camel?

[Joseph Kampf <jo...@gmail.com>]

If you store the messages in a DB, you can use the SQL component to drive your route.

You can craft your DB record to include a timestamp and have your SQL component use a relative time query so that once the record reaches the time you want it processed you can kick off your route.

Don’t forget to use the Idempotent consumer.  This is key when using the SQL component in an HA scenarios.

Joe




On 11/8/16, 11:24 AM, "Stephan Burkard" <sb...@gmail.com> wrote:

>@Brad: OK I guess that would work. But it's quite an implementation
>compared to the simple AMQ scheduler ;-)
>
>@James: Great question. I will definitely think about this. Have you got a
>generic approach to save/restore the message body and headers to/from the
>database?
>
>@Quinn: That's not a big issue in my case since the delay is 24 hours or
>less. The messages are currently in the queue, so the kaha-db-files cannot
>be deleted anyway.
>
>Once again thanks a lot for all your great answers!
>Stephan
>
>
>Quinn Stevenson <qu...@pronoia-solutions.com> schrieb am Fr., 4. Nov. 2016
>um 17:48 Uhr:
>
>> James brings up a very good point.
>>
>> I know ActiveMQ isn’t designed for long-term messages, and it’s bit me a
>> couple of times with KahaDB.  What happened was I had quite a few old
>> messages, but they were spaced out over time.  KahaDB doesn’t compact
>> files, and it can’t delete files until all the messages in the files are
>> processed.  Long story short - we ran out of disk.
>>
>> We addressed this part by using mKahaDB - it didn’t really “fix” the
>> problem, but the nature of our messaging is such that we could live with it.
>>
>> > On Nov 4, 2016, at 10:34 AM, James Green <ja...@gmail.com>
>> wrote:
>> >
>> > Are you unable to move the messages into a database for "long-term"
>> > storage? That's what we do, and we simply scan for messages due to be
>> > "re-sent" and re-submit them back into the queue.
>> >
>> > I do not believe ActiveMQ was really intended to hold many messages for
>> > long term storage, hence we use something that was.
>> >
>> >
>> > On 3 November 2016 at 15:46, Stephan Burkard <sb...@gmail.com> wrote:
>> >
>> >> Hi Camel users
>> >>
>> >> TLDR:
>> >> - how to implement wait/retry loop?
>> >> - messages "wait" in a queue
>> >> - each message should be reprocessed only once per interval (let's say
>> per
>> >> day)
>> >> - after reprocessing a message either returns to the waiting queue or
>> >> continues (goes out of loop)
>> >> - after [maxRetry] attempts it is moved away (goes out of loop)
>> >>
>> >>
>> >> Full story:
>> >>
>> >> As part of a typical processing workflow, I have a queue with "waiting"
>> >> messages. These messages are waiting for data of another system and I
>> have
>> >> no idea when the data arrives. Therefore I want to reprocess the
>> messages
>> >> in an interval, let's say once a day. If the data is still missing,
>> they go
>> >> again to the waiting queue.
>> >> To move messages away that would wait forever, I set and check a
>> maxRetry.
>> >>
>> >> How would I best implement this? Some thoughts I made:
>> >>
>> >> 1. Just throw an exception if the data is missing. The message is
>> >> redelivered.
>> >>   Problem: The message is instantly redelivered. So it does the retries
>> >> within milliseconds. I cannot customize the interval.
>> >>
>> >> 2. Use the Camel delayer.
>> >>   Works good in general (my current solution). I can even calculate the
>> >> delay based on the JMSTimestamp to take into account the time a message
>> >> waits to be consumed.
>> >>   Problem: For an interval of 1 day, a message is permanently in
>> >> processing state. The consumer is blocked. When I shutdown the
>> container,
>> >> Camel waits 5 minutes and after restarting the message goes to the DLQ.
>> >>
>> >> 3. Use a CronScheduledRoutePolicy to start/stop the route in the given
>> >> interval.
>> >>   Problem: When I start the route once a day for 5 minutes, the messages
>> >> with missing data are circulating until the route is stopped again. So
>> they
>> >> try to find the data multiple times per second and instantly reach the
>> >> maxRetry. They must try only once per interval.
>> >>
>> >> 4. Use a JMS selector as from-Endpoint that selects "JMSTimestamp <
>> >> (currentTime - intervalTime)"
>> >>   Problem: The from-Endpoint URI seems to be a constant, I can't make
>> the
>> >> selector dynamic.
>> >>
>> >> 5. Use the Camel Throttler
>> >>   Problem: The throttler sets a bandwith, that is not what we need.
>> >>
>> >> 6. Use a Timer from-Endpoint and hand over to a bean that consumes the
>> >> messages (newest idea, not yet tried).
>> >>   I guess I could manage to consume all waiting messages just once (even
>> >> if they already return before I am finished).
>> >>   Problems I can think of: It is not very nice to build another JMS
>> >> consumer (beside Camel), but I could use Spring JMS Template.
>> >>
>> >>
>> >> Numbers 1, 3, 4 and 5 simply do not work (or I don't use them the right
>> >> way) for this case. Number 2 is my current solution, number 6 a new
>> idea.
>> >> Any recommendations or other suggestions how to build this wait/retry
>> loop?
>> >>
>> >> Thanks a lot
>> >> Stephan
>> >>
>>
>>