You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by corcorda <da...@gmail.com> on 2007/09/04 16:39:44 UTC
Blocking transactions
I'm having some trouble with transactions. The problem is multiple consumers
block each other if they spend a long time in a transaction.
What I'm looking to do is send several hundred jobs into a queue. Each job
takes somewhere between a few seconds and 30 minutes to run. Each consumer
reads a job from the queue, processes it, writes the result back to the
queue then commits the transactions. If any problems happen in between it
should rollback and someone else should try the job. Each consumer runs in
it's own VM. It doesn't matter what order the jobs are processed in.
The problem is if one consumer receives a message and starts to process it
all the other consumer block on the receive call until the first one has
done a commit or rollback. This happens even though there are plenty of
messages in the queue. Is there a way around this?
Thanks.
--
View this message in context: http://www.nabble.com/Blocking-transactions-tf4378065s2354.html#a12479372
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Blocking transactions
Posted by Olivier OTTAVI <ol...@gmail.com>.
Thanks this was of great help also - I was wondering if their is a way to
set round robin load balancing policy on the client producers of a queue.
On 9/4/07, corcorda <da...@gmail.com> wrote:
>
>
> You were completely right. Sorry, I managed to read the prefetch article
> and
> still miss it.
>
> Thank you for your help.
>
>
> James.Strachan wrote:
> >
> > On 9/4/07, corcorda <da...@gmail.com> wrote:
> >>
> >> It's not the default prefetch values because I can recreate the problem
> >> with
> >> just 2 consumers and a few jobs.
> >
> > Thats normally exactly when the prefetch issue hits :)
> >
> http://activemq.apache.org/i-do-not-receive-messages-in-my-second-consumer.html
> >
> > Make sure you've 10,000 jobs for 2 consumers first - otherwise set
> > prefetch to 1.
> >
> >> The code is very simple. Create a connection using the
> >> ActiveMQConnectionFactory, call createSession with true and -1 as
> params.
> >> Then using this session create a producer that writes jobs and commits.
> >>
> >> On each consumer I do the exact same thing, except I create a consumer
> >> and
> >> call commit after the processing. They run in different VMs.
> >>
> >> What do you mean "Each consumer should be in its own session"? This
> >> sounds
> >> like where I'm going wrong ;)
> >
> > http://activemq.apache.org/how-do-i-use-jms-efficiently.html
> >
> > basically use 1 connection, then each consumer should be created from
> > its own session (so that it can consume its messages in parallel -
> > otherwise only one consumer will be active at any point in time).
> >
> >
> > --
> > James
> > -------
> > http://macstrac.blogspot.com/
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Blocking-transactions-tf4378065s2354.html#a12480568
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: Blocking transactions
Posted by corcorda <da...@gmail.com>.
You were completely right. Sorry, I managed to read the prefetch article and
still miss it.
Thank you for your help.
James.Strachan wrote:
>
> On 9/4/07, corcorda <da...@gmail.com> wrote:
>>
>> It's not the default prefetch values because I can recreate the problem
>> with
>> just 2 consumers and a few jobs.
>
> Thats normally exactly when the prefetch issue hits :)
> http://activemq.apache.org/i-do-not-receive-messages-in-my-second-consumer.html
>
> Make sure you've 10,000 jobs for 2 consumers first - otherwise set
> prefetch to 1.
>
>> The code is very simple. Create a connection using the
>> ActiveMQConnectionFactory, call createSession with true and -1 as params.
>> Then using this session create a producer that writes jobs and commits.
>>
>> On each consumer I do the exact same thing, except I create a consumer
>> and
>> call commit after the processing. They run in different VMs.
>>
>> What do you mean "Each consumer should be in its own session"? This
>> sounds
>> like where I'm going wrong ;)
>
> http://activemq.apache.org/how-do-i-use-jms-efficiently.html
>
> basically use 1 connection, then each consumer should be created from
> its own session (so that it can consume its messages in parallel -
> otherwise only one consumer will be active at any point in time).
>
>
> --
> James
> -------
> http://macstrac.blogspot.com/
>
>
--
View this message in context: http://www.nabble.com/Blocking-transactions-tf4378065s2354.html#a12480568
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Blocking transactions
Posted by James Strachan <ja...@gmail.com>.
On 9/4/07, corcorda <da...@gmail.com> wrote:
>
> It's not the default prefetch values because I can recreate the problem with
> just 2 consumers and a few jobs.
Thats normally exactly when the prefetch issue hits :)
http://activemq.apache.org/i-do-not-receive-messages-in-my-second-consumer.html
Make sure you've 10,000 jobs for 2 consumers first - otherwise set
prefetch to 1.
> The code is very simple. Create a connection using the
> ActiveMQConnectionFactory, call createSession with true and -1 as params.
> Then using this session create a producer that writes jobs and commits.
>
> On each consumer I do the exact same thing, except I create a consumer and
> call commit after the processing. They run in different VMs.
>
> What do you mean "Each consumer should be in its own session"? This sounds
> like where I'm going wrong ;)
http://activemq.apache.org/how-do-i-use-jms-efficiently.html
basically use 1 connection, then each consumer should be created from
its own session (so that it can consume its messages in parallel -
otherwise only one consumer will be active at any point in time).
--
James
-------
http://macstrac.blogspot.com/
Re: Blocking transactions
Posted by corcorda <da...@gmail.com>.
It's not the default prefetch values because I can recreate the problem with
just 2 consumers and a few jobs. I think the problem must be with the way
I'm using sessions. I haven't used JMS before so I'm following the samples
and maybe they're too simple.
The code is very simple. Create a connection using the
ActiveMQConnectionFactory, call createSession with true and -1 as params.
Then using this session create a producer that writes jobs and commits.
On each consumer I do the exact same thing, except I create a consumer and
call commit after the processing. They run in different VMs.
What do you mean "Each consumer should be in its own session"? This sounds
like where I'm going wrong ;)
James.Strachan wrote:
>
> On 9/4/07, corcorda <da...@gmail.com> wrote:
>>
>> I'm having some trouble with transactions. The problem is multiple
>> consumers
>> block each other if they spend a long time in a transaction.
>
> Each consumer should be in its own session to avoid them blocking each
> other right?
>
>
>> What I'm looking to do is send several hundred jobs into a queue. Each
>> job
>> takes somewhere between a few seconds and 30 minutes to run. Each
>> consumer
>> reads a job from the queue, processes it, writes the result back to the
>> queue then commits the transactions. If any problems happen in between it
>> should rollback and someone else should try the job. Each consumer runs
>> in
>> it's own VM. It doesn't matter what order the jobs are processed in.
>>
>> The problem is if one consumer receives a message and starts to process
>> it
>> all the other consumer block on the receive call until the first one has
>> done a commit or rollback. This happens even though there are plenty of
>> messages in the queue. Is there a way around this?
>
> Am thinking you're just hitting the default prefetch values...
> http://activemq.apache.org/what-is-the-prefetch-limit-for.html
>
> --
> James
> -------
> http://macstrac.blogspot.com/
>
>
--
View this message in context: http://www.nabble.com/Blocking-transactions-tf4378065s2354.html#a12480020
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Blocking transactions
Posted by James Strachan <ja...@gmail.com>.
On 9/4/07, corcorda <da...@gmail.com> wrote:
>
> I'm having some trouble with transactions. The problem is multiple consumers
> block each other if they spend a long time in a transaction.
Each consumer should be in its own session to avoid them blocking each
other right?
> What I'm looking to do is send several hundred jobs into a queue. Each job
> takes somewhere between a few seconds and 30 minutes to run. Each consumer
> reads a job from the queue, processes it, writes the result back to the
> queue then commits the transactions. If any problems happen in between it
> should rollback and someone else should try the job. Each consumer runs in
> it's own VM. It doesn't matter what order the jobs are processed in.
>
> The problem is if one consumer receives a message and starts to process it
> all the other consumer block on the receive call until the first one has
> done a commit or rollback. This happens even though there are plenty of
> messages in the queue. Is there a way around this?
Am thinking you're just hitting the default prefetch values...
http://activemq.apache.org/what-is-the-prefetch-limit-for.html
--
James
-------
http://macstrac.blogspot.com/