You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@camel.apache.org by boday <be...@initekconsulting.com> on 2013/04/03 05:02:22 UTC

Re: Transactional batching

this is a really old thread, but I'm wondering if anyone has any updated
ideas on this...I have a similar requirement to read from an AMQ queue,
batch X messages together and then process.  I'm currently doing this with
the aggregator/hawtdb repo combination but performance is proving too
slow...

I'd think it would be faster/cleaner to just be able to read multiple from
the queue in a transaction, aggregate together, process, then end the
transaction...

any ideas?


David J. M. Karlsen wrote
> Looking at this thread I wonder the same.
> 
> During a resource local jmstransaction you are able to fetch several
> messages - but there is no way in camel to control the commit size.
> You can do the same during a XA tx.
> 
> I see no support for batching here:
> http://camel.apache.org/batch-consumer.html.
> 
> Either transaction mode would benefit a lot if you could process several
> messages in one go.
> 
> Typical usecase would be tx( N messages, process them in paralell,
> aggregate and send out )





-----
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: http://camel.465427.n5.nabble.com/Transactional-batching-tp474002p5730245.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Transactional batching

Posted by boday <be...@initekconsulting.com>.

thanks Scott, I noticed that SJMS supports batched Tx, but given that it
hasn't been released yet, I assume others have had to work around this issue
using camle-jms in the past...sounds like I'll need to roll my own or rely
on aggregator/hawtdb EIP until 2.11 can be used...


sully6768 wrote
> Hi Ben,
> 
> Currently only SJMS (2.11.0) has support for batching of JMS transactions
> (local only, no XA).  The consumer has natural batch support meaning that
> you set the number of messages you would like read before issuing a
> commit.
>  You can also specify a timeout, default is 5000ms, to keep the consumer
> from holding the transaction open if you have a bursty message flow.
> 
> On the producer side SJMS is able to support using a List of Objects or
> SJMS BatchMessage objects where the producer will iterate through the list
> and if configured for transactions, commit or rollback the transaction.
>  This is necessary at this time since using a batch on the producer would
> potentially cause the initial consumer to loose visibility of what
> messages
> have been lost in the event of a rollback.  This would work well when
> using
> a ProducerTemplate in an application that collects a given number of
> messages and then publishes the collection.
> 
> There currently isn't support for XA but that can be mitigated by having
> applications monitor for the JMSRedelivery=true header.
> 
> Let me know if this answers your question.
> 
> Best Regards,
> Scott ES
> 
> 
> On Tue, Apr 2, 2013 at 11:02 PM, boday &lt;

> ben.oday@

> &gt;wrote:
> 
>> this is a really old thread, but I'm wondering if anyone has any updated
>> ideas on this...I have a similar requirement to read from an AMQ queue,
>> batch X messages together and then process.  I'm currently doing this
>> with
>> the aggregator/hawtdb repo combination but performance is proving too
>> slow...
>>
>> I'd think it would be faster/cleaner to just be able to read multiple
>> from
>> the queue in a transaction, aggregate together, process, then end the
>> transaction...
>>
>> any ideas?
>>
>>
>> David J. M. Karlsen wrote
>> > Looking at this thread I wonder the same.
>> >
>> > During a resource local jmstransaction you are able to fetch several
>> > messages - but there is no way in camel to control the commit size.
>> > You can do the same during a XA tx.
>> >
>> > I see no support for batching here:
>> > http://camel.apache.org/batch-consumer.html.
>> >
>> > Either transaction mode would benefit a lot if you could process
>> several
>> > messages in one go.
>> >
>> > Typical usecase would be tx( N messages, process them in paralell,
>> > aggregate and send out )
>>
>>
>>
>>
>>
>> -----
>> Ben O'Day
>> IT Consultant -http://consulting-notes.com
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Transactional-batching-tp474002p5730245.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
> 
> 
> 
> -- 
> -- 
> Scott England-Sullivan
> Apache Camel Committer
> Principal Consultant / Sr. Architect | Red Hat, Inc.
> FuseSource is now part of Red Hat
> Web:     fusesource.com &lt;http://www.fusesource.com&gt; |
> redhat.com&lt;http://www.redhat.com&gt;
> Blog:     sully6768.blogspot.com
> Twitter: sully6768





-----
Ben O'Day
IT Consultant -http://consulting-notes.com

--
View this message in context: http://camel.465427.n5.nabble.com/Transactional-batching-tp474002p5730351.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Transactional batching

Posted by Scott England-Sullivan <su...@gmail.com>.

Hi Ben,

Currently only SJMS (2.11.0) has support for batching of JMS transactions
(local only, no XA).  The consumer has natural batch support meaning that
you set the number of messages you would like read before issuing a commit.
 You can also specify a timeout, default is 5000ms, to keep the consumer
from holding the transaction open if you have a bursty message flow.

On the producer side SJMS is able to support using a List of Objects or
SJMS BatchMessage objects where the producer will iterate through the list
and if configured for transactions, commit or rollback the transaction.
 This is necessary at this time since using a batch on the producer would
potentially cause the initial consumer to loose visibility of what messages
have been lost in the event of a rollback.  This would work well when using
a ProducerTemplate in an application that collects a given number of
messages and then publishes the collection.

There currently isn't support for XA but that can be mitigated by having
applications monitor for the JMSRedelivery=true header.

Let me know if this answers your question.

Best Regards,
Scott ES

On Tue, Apr 2, 2013 at 11:02 PM, boday <be...@initekconsulting.com>wrote:

> this is a really old thread, but I'm wondering if anyone has any updated
> ideas on this...I have a similar requirement to read from an AMQ queue,
> batch X messages together and then process.  I'm currently doing this with
> the aggregator/hawtdb repo combination but performance is proving too
> slow...
>
> I'd think it would be faster/cleaner to just be able to read multiple from
> the queue in a transaction, aggregate together, process, then end the
> transaction...
>
> any ideas?
>
>
> David J. M. Karlsen wrote
> > Looking at this thread I wonder the same.
> >
> > During a resource local jmstransaction you are able to fetch several
> > messages - but there is no way in camel to control the commit size.
> > You can do the same during a XA tx.
> >
> > I see no support for batching here:
> > http://camel.apache.org/batch-consumer.html.
> >
> > Either transaction mode would benefit a lot if you could process several
> > messages in one go.
> >
> > Typical usecase would be tx( N messages, process them in paralell,
> > aggregate and send out )
>
>
>
>
>
> -----
> Ben O'Day
> IT Consultant -http://consulting-notes.com
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Transactional-batching-tp474002p5730245.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

-- 
-- 
Scott England-Sullivan
Apache Camel Committer
Principal Consultant / Sr. Architect | Red Hat, Inc.
FuseSource is now part of Red Hat
Web:     fusesource.com <http://www.fusesource.com> |
redhat.com<http://www.redhat.com>
Blog:     sully6768.blogspot.com
Twitter: sully6768