Camel Quartz Query

[gilboy <jo...@gmail.com>]

Hi 

I have a quartz job(stateful) which wakes up every 60 seconds, pulls
"unprocessed" records from a DB and sends each of them to a Web Service
Endpoint. The Web Service call introduces a latency of approx .5 sec. Each
time the quartz job fires there can be 100's of "unprocessed" records.
Hence, after retrieving the records I iterate over the records via the
splitter and I load balance(sticky - need to maintain order across subsets
of the records) to 10 different seda queues. I then have 10 different routes
which takes from each of the queues, transmits the message to the web
service endpoint and then updates the record to be in a "processed" state.

However, with the above approach I could easily send duplicates to the web
service endpoint because a record which is returned when the quartz job
fires at time 0 could be sitting on one of the seda queues at time 60 when
the quartz job fires again and hence it is still in an "unprocessed" state
in the database as it only gets updated to "processed" when we get a
response from the Web Service call.

I could solve this by introducing a new interim status and have the DB
update the "unprocessed" records to the new interim status but just wanted
to check to see if there was any other way of solving this as I have less
control over the DB

Thanks
Joe



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-Quartz-Query-tp5740206.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel Quartz Query

[Christian Posta <ch...@gmail.com>]

+1 idempotent consumer. that should work nicely.


On Wed, Sep 25, 2013 at 10:59 PM, Taariq Levack <ta...@gmail.com> wrote:

> Hi
>
> Can you use an idempotent consumer[1] before sending to the web service?
> http://camel.apache.org/idempotent-consumer.html
>
> If you did have a sort of 'busy' status, you'd have to handle the situation
> where records are left in that 'busy' state due to a JVM crash for example.
>
> Taariq
>
>
> On Wed, Sep 25, 2013 at 5:20 PM, gilboy <jo...@gmail.com> wrote:
>
> > Hi
> >
> > I have a quartz job(stateful) which wakes up every 60 seconds, pulls
> > "unprocessed" records from a DB and sends each of them to a Web Service
> > Endpoint. The Web Service call introduces a latency of approx .5 sec.
> Each
> > time the quartz job fires there can be 100's of "unprocessed" records.
> > Hence, after retrieving the records I iterate over the records via the
> > splitter and I load balance(sticky - need to maintain order across
> subsets
> > of the records) to 10 different seda queues. I then have 10 different
> > routes
> > which takes from each of the queues, transmits the message to the web
> > service endpoint and then updates the record to be in a "processed"
> state.
> >
> > However, with the above approach I could easily send duplicates to the
> web
> > service endpoint because a record which is returned when the quartz job
> > fires at time 0 could be sitting on one of the seda queues at time 60
> when
> > the quartz job fires again and hence it is still in an "unprocessed"
> state
> > in the database as it only gets updated to "processed" when we get a
> > response from the Web Service call.
> >
> > I could solve this by introducing a new interim status and have the DB
> > update the "unprocessed" records to the new interim status but just
> wanted
> > to check to see if there was any other way of solving this as I have less
> > control over the DB
> >
> > Thanks
> > Joe
> >
> >
> >
> > --
> > View this message in context:
> > http://camel.465427.n5.nabble.com/Camel-Quartz-Query-tp5740206.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
> >
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Re: Camel Quartz Query

[Taariq Levack <ta...@gmail.com>]

Hi

Can you use an idempotent consumer[1] before sending to the web service?
http://camel.apache.org/idempotent-consumer.html

If you did have a sort of 'busy' status, you'd have to handle the situation
where records are left in that 'busy' state due to a JVM crash for example.

Taariq


On Wed, Sep 25, 2013 at 5:20 PM, gilboy <jo...@gmail.com> wrote:

> Hi
>
> I have a quartz job(stateful) which wakes up every 60 seconds, pulls
> "unprocessed" records from a DB and sends each of them to a Web Service
> Endpoint. The Web Service call introduces a latency of approx .5 sec. Each
> time the quartz job fires there can be 100's of "unprocessed" records.
> Hence, after retrieving the records I iterate over the records via the
> splitter and I load balance(sticky - need to maintain order across subsets
> of the records) to 10 different seda queues. I then have 10 different
> routes
> which takes from each of the queues, transmits the message to the web
> service endpoint and then updates the record to be in a "processed" state.
>
> However, with the above approach I could easily send duplicates to the web
> service endpoint because a record which is returned when the quartz job
> fires at time 0 could be sitting on one of the seda queues at time 60 when
> the quartz job fires again and hence it is still in an "unprocessed" state
> in the database as it only gets updated to "processed" when we get a
> response from the Web Service call.
>
> I could solve this by introducing a new interim status and have the DB
> update the "unprocessed" records to the new interim status but just wanted
> to check to see if there was any other way of solving this as I have less
> control over the DB
>
> Thanks
> Joe
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Camel-Quartz-Query-tp5740206.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>