You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by rodos77 <eu...@nexj.com> on 2010/03/18 23:05:38 UTC
Re: prefetchExtension off-by-1 for transacted consumers with
prefetchSize > 0
I don't really get this. If you extend the prefetchSize to more than what it
was configured to be by the maxMessages parameter in the
Connection.createConnectionConsumer() call, you are violating the JMS spec.
The maxMessages parameter is defined as "the maximum number of messages that
can be assigned to a server session at one time", so how can you make it
bigger than what it was set to by the ConnectionConsumer?!
ActiveMQ Resource Adapter has 2 parameters in its ActivationSpec that
control the maxMessages argument used in the
Connection.createConnectionConsumer() call: maxSessions and
maxMessagesPerSessions. The maxMessages argument is set to their product.
This makes sense. If a user configures ActiveMQ to have x maxSessions and y
maxMessagesPerSessions, maxMessages (and consequentially the prefetchSize)
should be set to x*y. But why should this ever be extended? Shouldn't it
be expected that only a maximum of x*y messages be delivered at the same
time? If a user wants to process z <= x*y messages in the same tx, that
should not pose any problem as the RA will have x ServerSessions each
capable of processing y messages at the same time. But if the user wants to
process z > x*y messages at the same time after explicitly configuring
ActiveMQ to only deliver a maximum of x*y messages at the same time, this is
a user error/problem. ActiveMQ should not be violating the JMS Spec to
accommodate this. What is the point of having those properties in the 1st
place if you are going to extend them and deliver as many messages as
available?
Gary Tully wrote:
>
> The prefetch extension in the transacted case is necessary for the case
> where a transacted consumer wants to consumer more than prefetch messages
> in
> a single transaction. Without the extension, the broker would see a full
> prefetch and not dispatch any more and a receive could block.
>
> On 15 March 2010 21:31, rodos77 <eu...@nexj.com> wrote:
>
>>
>> JIRA AMQ-2651 opened.
>>
>> I would be happy to provide a patch but as I stated in my original post,
>> I
>> don't fully understand the need for prefetchExtension in the case of a
>> transacted, asynchronous (prefetchSize > 0) consumer. If somebody could
>> explain it, that would perhaps give me the required knowledge to come up
>> with a patch.
>>
>> The way I see it now, I would get rid of the prefetchExtension (in the
>> above
>> mentioned case) all together, but again I'm probably missing some piece
>> of
>> information and I don't want to cause any side effects. Barring that, I
>> think there is an off-by-1 error in its calculation but I need this
>> confirmed in order to create a proper patch.
>> --
>> View this message in context:
>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27910673.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27950855.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: prefetchExtension off-by-1 for transacted consumers with
prefetchSize > 0
Posted by rodos77 <eu...@nexj.com>.
So I finally got around to doing this. I've made changes on a per
destination basis only for now and have attached the patch to the JIRA.
Also, can you please comment on
https://issues.apache.org/activemq/browse/AMQ-2659 AMQ-2659 ? I've proposed
a small code change for that. Should be a quick fix. We need this as well
for our stuff to work.
And of course, there are still the issues
https://issues.apache.org/activemq/browse/AMQ-2652 AMQ-2652 and
https://issues.apache.org/activemq/browse/AMQ-2653 AMQ-2653 . I know those
are much broader and are harder to fix but I just wanted to know if there
are plans.
Thx!
rodos77 wrote:
>
> Ok, I have to finish some work first but I'll see how much I can get done
> after that. I'm not too familiar with the classes you mentioned but I'll
> see what I can figure out.
>
> In principle, I can wait till 5.4 but I'll see what I can do.
>
> There is no problem with AMQ RA, it's just that we have our own JMS RA
> that is capable of integrating with any JMS engine among other features.
> We use this RA in our enterprise stack.
>
>
>
> Gary Tully wrote:
>>
>> If you are in the thick of it and can propose a patch that would be
>> great.
>> A new boolean option on ActiveMQPolicy would be the best way to introduce
>> the configuration on a destination basis (set in
>> org.apache.activemq.broker.region.policy.PolicyEntry.configure(Broker,
>> SystemUsage, QueueSubscription) and if you want more fine grain control,
>> an
>> option can be added to ConsumerInfo to allow the configuration on a per
>> subscription basis, propagating the change though to a connection and
>> connection factory. That will require regenerating the wireformat
>> marshallers (in activemq-core; mvn -Popenwire-generate generate-sources)
>> so
>> that the new option is propagated from the client. See how you get on.
>>
>> Otherwise, we can try and get this change into the 5.4 release.
>>
>> BTW: Is there a problem with the activemq RA that stops you from using
>> it?
>>
>>
>> On 22 March 2010 14:56, rodos77 <eu...@nexj.com> wrote:
>>
>>>
>>> I would point out that another JIRA that I've opened (
>>> https://issues.apache.org/activemq/browse/AMQ-2652 AMQ-2652 ),
>>> illustrates
>>> a
>>> deadlock (hang) which is triggered precisely by the prefetch extension
>>> of
>>> transacted consumers (see JIRA for details). So while trying to avoid
>>> one
>>> potential hang, the use of prefetch extension causes another.
>>>
>>> However, the solution that you propose to make the prefetch extension
>>> configurable and to have it off by default sounds good to me. How
>>> should
>>> we
>>> proceed to have this put in place?
>>>
>>>
>>>
>>> Gary Tully wrote:
>>> >
>>> > Fair point. From the perspective of the ResourceAdapter where there
>>> are
>>> > explicit expectations this makes good sense. The prefetch should be
>>> > written
>>> > in stone.
>>> >
>>> > From a regular transacted consumers perspective, where the prefetch is
>>> > largely hidden, avoiding a hang for large transactions is good because
>>> a
>>> > hang would be hard to diagnose. I agree it could be considered
>>> user/error
>>> > but it is difficult to flag it as such.
>>> >
>>> > It may make sense to make the (extend the prefetch for a
>>> > transaction) behavior configurable such that it can be disabled for
>>> the
>>> RA
>>> > or off by default and enabled in the event that large transactions
>>> hang.
>>> > Thinking more, off by default is probably the best approach as
>>> > transactions
>>> > that exceed the prefetch should be rare and the RAR behavior would
>>> match
>>> > expectations.
>>> >
>>> > On 18 March 2010 22:05, rodos77 <eu...@nexj.com> wrote:
>>> >
>>> >>
>>> >> I don't really get this. If you extend the prefetchSize to more than
>>> >> what
>>> >> it
>>> >> was configured to be by the maxMessages parameter in the
>>> >> Connection.createConnectionConsumer() call, you are violating the JMS
>>> >> spec.
>>> >> The maxMessages parameter is defined as "the maximum number of
>>> messages
>>> >> that
>>> >> can be assigned to a server session at one time", so how can you make
>>> it
>>> >> bigger than what it was set to by the ConnectionConsumer?!
>>> >>
>>> >> ActiveMQ Resource Adapter has 2 parameters in its ActivationSpec that
>>> >> control the maxMessages argument used in the
>>> >> Connection.createConnectionConsumer() call: maxSessions and
>>> >> maxMessagesPerSessions. The maxMessages argument is set to their
>>> >> product.
>>> >> This makes sense. If a user configures ActiveMQ to have x
>>> maxSessions
>>> >> and
>>> >> y
>>> >> maxMessagesPerSessions, maxMessages (and consequentially the
>>> >> prefetchSize)
>>> >> should be set to x*y. But why should this ever be extended?
>>> Shouldn't
>>> >> it
>>> >> be expected that only a maximum of x*y messages be delivered at the
>>> same
>>> >> time? If a user wants to process z <= x*y messages in the same tx,
>>> that
>>> >> should not pose any problem as the RA will have x ServerSessions each
>>> >> capable of processing y messages at the same time. But if the user
>>> wants
>>> >> to
>>> >> process z > x*y messages at the same time after explicitly
>>> configuring
>>> >> ActiveMQ to only deliver a maximum of x*y messages at the same time,
>>> this
>>> >> is
>>> >> a user error/problem. ActiveMQ should not be violating the JMS Spec
>>> to
>>> >> accommodate this. What is the point of having those properties in
>>> the
>>> >> 1st
>>> >> place if you are going to extend them and deliver as many messages as
>>> >> available?
>>> >>
>>> >>
>>> >>
>>> >> Gary Tully wrote:
>>> >> >
>>> >> > The prefetch extension in the transacted case is necessary for the
>>> case
>>> >> > where a transacted consumer wants to consumer more than prefetch
>>> >> messages
>>> >> > in
>>> >> > a single transaction. Without the extension, the broker would see a
>>> >> full
>>> >> > prefetch and not dispatch any more and a receive could block.
>>> >> >
>>> >> > On 15 March 2010 21:31, rodos77 <eu...@nexj.com> wrote:
>>> >> >
>>> >> >>
>>> >> >> JIRA AMQ-2651 opened.
>>> >> >>
>>> >> >> I would be happy to provide a patch but as I stated in my original
>>> >> post,
>>> >> >> I
>>> >> >> don't fully understand the need for prefetchExtension in the case
>>> of
>>> a
>>> >> >> transacted, asynchronous (prefetchSize > 0) consumer. If somebody
>>> >> could
>>> >> >> explain it, that would perhaps give me the required knowledge to
>>> come
>>> >> up
>>> >> >> with a patch.
>>> >> >>
>>> >> >> The way I see it now, I would get rid of the prefetchExtension (in
>>> the
>>> >> >> above
>>> >> >> mentioned case) all together, but again I'm probably missing some
>>> >> piece
>>> >> >> of
>>> >> >> information and I don't want to cause any side effects. Barring
>>> that,
>>> >> I
>>> >> >> think there is an off-by-1 error in its calculation but I need
>>> this
>>> >> >> confirmed in order to create a proper patch.
>>> >> >> --
>>> >> >> View this message in context:
>>> >> >>
>>> >>
>>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27910673.html
>>> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>> >> >>
>>> >> >>
>>> >> >
>>> >> >
>>> >> > --
>>> >> > http://blog.garytully.com
>>> >> >
>>> >> > Open Source Integration
>>> >> > http://fusesource.com
>>> >> >
>>> >> >
>>> >>
>>> >> --
>>> >> View this message in context:
>>> >>
>>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27950855.html
>>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > http://blog.garytully.com
>>> >
>>> > Open Source Integration
>>> > http://fusesource.com
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27987505.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source Integration
>> http://fusesource.com
>>
>>
>
>
--
View this message in context: http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p28257116.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: prefetchExtension off-by-1 for transacted consumers with
prefetchSize > 0
Posted by rodos77 <eu...@nexj.com>.
Ok, I have to finish some work first but I'll see how much I can get done
after that. I'm not too familiar with the classes you mentioned but I'll
see what I can figure out.
In principle, I can wait till 5.4 but I'll see what I can do.
There is no problem with AMQ RA, it's just that we have our own JMS RA that
is capable of integrating with any JMS engine among other features. We use
this RA in our enterprise stack.
Gary Tully wrote:
>
> If you are in the thick of it and can propose a patch that would be great.
> A new boolean option on ActiveMQPolicy would be the best way to introduce
> the configuration on a destination basis (set in
> org.apache.activemq.broker.region.policy.PolicyEntry.configure(Broker,
> SystemUsage, QueueSubscription) and if you want more fine grain control,
> an
> option can be added to ConsumerInfo to allow the configuration on a per
> subscription basis, propagating the change though to a connection and
> connection factory. That will require regenerating the wireformat
> marshallers (in activemq-core; mvn -Popenwire-generate generate-sources)
> so
> that the new option is propagated from the client. See how you get on.
>
> Otherwise, we can try and get this change into the 5.4 release.
>
> BTW: Is there a problem with the activemq RA that stops you from using it?
>
>
> On 22 March 2010 14:56, rodos77 <eu...@nexj.com> wrote:
>
>>
>> I would point out that another JIRA that I've opened (
>> https://issues.apache.org/activemq/browse/AMQ-2652 AMQ-2652 ),
>> illustrates
>> a
>> deadlock (hang) which is triggered precisely by the prefetch extension of
>> transacted consumers (see JIRA for details). So while trying to avoid
>> one
>> potential hang, the use of prefetch extension causes another.
>>
>> However, the solution that you propose to make the prefetch extension
>> configurable and to have it off by default sounds good to me. How should
>> we
>> proceed to have this put in place?
>>
>>
>>
>> Gary Tully wrote:
>> >
>> > Fair point. From the perspective of the ResourceAdapter where there are
>> > explicit expectations this makes good sense. The prefetch should be
>> > written
>> > in stone.
>> >
>> > From a regular transacted consumers perspective, where the prefetch is
>> > largely hidden, avoiding a hang for large transactions is good because
>> a
>> > hang would be hard to diagnose. I agree it could be considered
>> user/error
>> > but it is difficult to flag it as such.
>> >
>> > It may make sense to make the (extend the prefetch for a
>> > transaction) behavior configurable such that it can be disabled for the
>> RA
>> > or off by default and enabled in the event that large transactions
>> hang.
>> > Thinking more, off by default is probably the best approach as
>> > transactions
>> > that exceed the prefetch should be rare and the RAR behavior would
>> match
>> > expectations.
>> >
>> > On 18 March 2010 22:05, rodos77 <eu...@nexj.com> wrote:
>> >
>> >>
>> >> I don't really get this. If you extend the prefetchSize to more than
>> >> what
>> >> it
>> >> was configured to be by the maxMessages parameter in the
>> >> Connection.createConnectionConsumer() call, you are violating the JMS
>> >> spec.
>> >> The maxMessages parameter is defined as "the maximum number of
>> messages
>> >> that
>> >> can be assigned to a server session at one time", so how can you make
>> it
>> >> bigger than what it was set to by the ConnectionConsumer?!
>> >>
>> >> ActiveMQ Resource Adapter has 2 parameters in its ActivationSpec that
>> >> control the maxMessages argument used in the
>> >> Connection.createConnectionConsumer() call: maxSessions and
>> >> maxMessagesPerSessions. The maxMessages argument is set to their
>> >> product.
>> >> This makes sense. If a user configures ActiveMQ to have x maxSessions
>> >> and
>> >> y
>> >> maxMessagesPerSessions, maxMessages (and consequentially the
>> >> prefetchSize)
>> >> should be set to x*y. But why should this ever be extended?
>> Shouldn't
>> >> it
>> >> be expected that only a maximum of x*y messages be delivered at the
>> same
>> >> time? If a user wants to process z <= x*y messages in the same tx,
>> that
>> >> should not pose any problem as the RA will have x ServerSessions each
>> >> capable of processing y messages at the same time. But if the user
>> wants
>> >> to
>> >> process z > x*y messages at the same time after explicitly configuring
>> >> ActiveMQ to only deliver a maximum of x*y messages at the same time,
>> this
>> >> is
>> >> a user error/problem. ActiveMQ should not be violating the JMS Spec
>> to
>> >> accommodate this. What is the point of having those properties in the
>> >> 1st
>> >> place if you are going to extend them and deliver as many messages as
>> >> available?
>> >>
>> >>
>> >>
>> >> Gary Tully wrote:
>> >> >
>> >> > The prefetch extension in the transacted case is necessary for the
>> case
>> >> > where a transacted consumer wants to consumer more than prefetch
>> >> messages
>> >> > in
>> >> > a single transaction. Without the extension, the broker would see a
>> >> full
>> >> > prefetch and not dispatch any more and a receive could block.
>> >> >
>> >> > On 15 March 2010 21:31, rodos77 <eu...@nexj.com> wrote:
>> >> >
>> >> >>
>> >> >> JIRA AMQ-2651 opened.
>> >> >>
>> >> >> I would be happy to provide a patch but as I stated in my original
>> >> post,
>> >> >> I
>> >> >> don't fully understand the need for prefetchExtension in the case
>> of
>> a
>> >> >> transacted, asynchronous (prefetchSize > 0) consumer. If somebody
>> >> could
>> >> >> explain it, that would perhaps give me the required knowledge to
>> come
>> >> up
>> >> >> with a patch.
>> >> >>
>> >> >> The way I see it now, I would get rid of the prefetchExtension (in
>> the
>> >> >> above
>> >> >> mentioned case) all together, but again I'm probably missing some
>> >> piece
>> >> >> of
>> >> >> information and I don't want to cause any side effects. Barring
>> that,
>> >> I
>> >> >> think there is an off-by-1 error in its calculation but I need this
>> >> >> confirmed in order to create a proper patch.
>> >> >> --
>> >> >> View this message in context:
>> >> >>
>> >>
>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27910673.html
>> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> > --
>> >> > http://blog.garytully.com
>> >> >
>> >> > Open Source Integration
>> >> > http://fusesource.com
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27950855.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source Integration
>> > http://fusesource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27987505.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27988380.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: prefetchExtension off-by-1 for transacted consumers with
prefetchSize > 0
Posted by Gary Tully <ga...@gmail.com>.
If you are in the thick of it and can propose a patch that would be great.
A new boolean option on ActiveMQPolicy would be the best way to introduce
the configuration on a destination basis (set in
org.apache.activemq.broker.region.policy.PolicyEntry.configure(Broker,
SystemUsage, QueueSubscription) and if you want more fine grain control, an
option can be added to ConsumerInfo to allow the configuration on a per
subscription basis, propagating the change though to a connection and
connection factory. That will require regenerating the wireformat
marshallers (in activemq-core; mvn -Popenwire-generate generate-sources) so
that the new option is propagated from the client. See how you get on.
Otherwise, we can try and get this change into the 5.4 release.
BTW: Is there a problem with the activemq RA that stops you from using it?
On 22 March 2010 14:56, rodos77 <eu...@nexj.com> wrote:
>
> I would point out that another JIRA that I've opened (
> https://issues.apache.org/activemq/browse/AMQ-2652 AMQ-2652 ), illustrates
> a
> deadlock (hang) which is triggered precisely by the prefetch extension of
> transacted consumers (see JIRA for details). So while trying to avoid one
> potential hang, the use of prefetch extension causes another.
>
> However, the solution that you propose to make the prefetch extension
> configurable and to have it off by default sounds good to me. How should
> we
> proceed to have this put in place?
>
>
>
> Gary Tully wrote:
> >
> > Fair point. From the perspective of the ResourceAdapter where there are
> > explicit expectations this makes good sense. The prefetch should be
> > written
> > in stone.
> >
> > From a regular transacted consumers perspective, where the prefetch is
> > largely hidden, avoiding a hang for large transactions is good because a
> > hang would be hard to diagnose. I agree it could be considered user/error
> > but it is difficult to flag it as such.
> >
> > It may make sense to make the (extend the prefetch for a
> > transaction) behavior configurable such that it can be disabled for the
> RA
> > or off by default and enabled in the event that large transactions hang.
> > Thinking more, off by default is probably the best approach as
> > transactions
> > that exceed the prefetch should be rare and the RAR behavior would match
> > expectations.
> >
> > On 18 March 2010 22:05, rodos77 <eu...@nexj.com> wrote:
> >
> >>
> >> I don't really get this. If you extend the prefetchSize to more than
> >> what
> >> it
> >> was configured to be by the maxMessages parameter in the
> >> Connection.createConnectionConsumer() call, you are violating the JMS
> >> spec.
> >> The maxMessages parameter is defined as "the maximum number of messages
> >> that
> >> can be assigned to a server session at one time", so how can you make it
> >> bigger than what it was set to by the ConnectionConsumer?!
> >>
> >> ActiveMQ Resource Adapter has 2 parameters in its ActivationSpec that
> >> control the maxMessages argument used in the
> >> Connection.createConnectionConsumer() call: maxSessions and
> >> maxMessagesPerSessions. The maxMessages argument is set to their
> >> product.
> >> This makes sense. If a user configures ActiveMQ to have x maxSessions
> >> and
> >> y
> >> maxMessagesPerSessions, maxMessages (and consequentially the
> >> prefetchSize)
> >> should be set to x*y. But why should this ever be extended? Shouldn't
> >> it
> >> be expected that only a maximum of x*y messages be delivered at the same
> >> time? If a user wants to process z <= x*y messages in the same tx, that
> >> should not pose any problem as the RA will have x ServerSessions each
> >> capable of processing y messages at the same time. But if the user
> wants
> >> to
> >> process z > x*y messages at the same time after explicitly configuring
> >> ActiveMQ to only deliver a maximum of x*y messages at the same time,
> this
> >> is
> >> a user error/problem. ActiveMQ should not be violating the JMS Spec to
> >> accommodate this. What is the point of having those properties in the
> >> 1st
> >> place if you are going to extend them and deliver as many messages as
> >> available?
> >>
> >>
> >>
> >> Gary Tully wrote:
> >> >
> >> > The prefetch extension in the transacted case is necessary for the
> case
> >> > where a transacted consumer wants to consumer more than prefetch
> >> messages
> >> > in
> >> > a single transaction. Without the extension, the broker would see a
> >> full
> >> > prefetch and not dispatch any more and a receive could block.
> >> >
> >> > On 15 March 2010 21:31, rodos77 <eu...@nexj.com> wrote:
> >> >
> >> >>
> >> >> JIRA AMQ-2651 opened.
> >> >>
> >> >> I would be happy to provide a patch but as I stated in my original
> >> post,
> >> >> I
> >> >> don't fully understand the need for prefetchExtension in the case of
> a
> >> >> transacted, asynchronous (prefetchSize > 0) consumer. If somebody
> >> could
> >> >> explain it, that would perhaps give me the required knowledge to come
> >> up
> >> >> with a patch.
> >> >>
> >> >> The way I see it now, I would get rid of the prefetchExtension (in
> the
> >> >> above
> >> >> mentioned case) all together, but again I'm probably missing some
> >> piece
> >> >> of
> >> >> information and I don't want to cause any side effects. Barring
> that,
> >> I
> >> >> think there is an off-by-1 error in its calculation but I need this
> >> >> confirmed in order to create a proper patch.
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27910673.html
> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > http://blog.garytully.com
> >> >
> >> > Open Source Integration
> >> > http://fusesource.com
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27950855.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27987505.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: prefetchExtension off-by-1 for transacted consumers with
prefetchSize > 0
Posted by rodos77 <eu...@nexj.com>.
I would point out that another JIRA that I've opened (
https://issues.apache.org/activemq/browse/AMQ-2652 AMQ-2652 ), illustrates a
deadlock (hang) which is triggered precisely by the prefetch extension of
transacted consumers (see JIRA for details). So while trying to avoid one
potential hang, the use of prefetch extension causes another.
However, the solution that you propose to make the prefetch extension
configurable and to have it off by default sounds good to me. How should we
proceed to have this put in place?
Gary Tully wrote:
>
> Fair point. From the perspective of the ResourceAdapter where there are
> explicit expectations this makes good sense. The prefetch should be
> written
> in stone.
>
> From a regular transacted consumers perspective, where the prefetch is
> largely hidden, avoiding a hang for large transactions is good because a
> hang would be hard to diagnose. I agree it could be considered user/error
> but it is difficult to flag it as such.
>
> It may make sense to make the (extend the prefetch for a
> transaction) behavior configurable such that it can be disabled for the RA
> or off by default and enabled in the event that large transactions hang.
> Thinking more, off by default is probably the best approach as
> transactions
> that exceed the prefetch should be rare and the RAR behavior would match
> expectations.
>
> On 18 March 2010 22:05, rodos77 <eu...@nexj.com> wrote:
>
>>
>> I don't really get this. If you extend the prefetchSize to more than
>> what
>> it
>> was configured to be by the maxMessages parameter in the
>> Connection.createConnectionConsumer() call, you are violating the JMS
>> spec.
>> The maxMessages parameter is defined as "the maximum number of messages
>> that
>> can be assigned to a server session at one time", so how can you make it
>> bigger than what it was set to by the ConnectionConsumer?!
>>
>> ActiveMQ Resource Adapter has 2 parameters in its ActivationSpec that
>> control the maxMessages argument used in the
>> Connection.createConnectionConsumer() call: maxSessions and
>> maxMessagesPerSessions. The maxMessages argument is set to their
>> product.
>> This makes sense. If a user configures ActiveMQ to have x maxSessions
>> and
>> y
>> maxMessagesPerSessions, maxMessages (and consequentially the
>> prefetchSize)
>> should be set to x*y. But why should this ever be extended? Shouldn't
>> it
>> be expected that only a maximum of x*y messages be delivered at the same
>> time? If a user wants to process z <= x*y messages in the same tx, that
>> should not pose any problem as the RA will have x ServerSessions each
>> capable of processing y messages at the same time. But if the user wants
>> to
>> process z > x*y messages at the same time after explicitly configuring
>> ActiveMQ to only deliver a maximum of x*y messages at the same time, this
>> is
>> a user error/problem. ActiveMQ should not be violating the JMS Spec to
>> accommodate this. What is the point of having those properties in the
>> 1st
>> place if you are going to extend them and deliver as many messages as
>> available?
>>
>>
>>
>> Gary Tully wrote:
>> >
>> > The prefetch extension in the transacted case is necessary for the case
>> > where a transacted consumer wants to consumer more than prefetch
>> messages
>> > in
>> > a single transaction. Without the extension, the broker would see a
>> full
>> > prefetch and not dispatch any more and a receive could block.
>> >
>> > On 15 March 2010 21:31, rodos77 <eu...@nexj.com> wrote:
>> >
>> >>
>> >> JIRA AMQ-2651 opened.
>> >>
>> >> I would be happy to provide a patch but as I stated in my original
>> post,
>> >> I
>> >> don't fully understand the need for prefetchExtension in the case of a
>> >> transacted, asynchronous (prefetchSize > 0) consumer. If somebody
>> could
>> >> explain it, that would perhaps give me the required knowledge to come
>> up
>> >> with a patch.
>> >>
>> >> The way I see it now, I would get rid of the prefetchExtension (in the
>> >> above
>> >> mentioned case) all together, but again I'm probably missing some
>> piece
>> >> of
>> >> information and I don't want to cause any side effects. Barring that,
>> I
>> >> think there is an off-by-1 error in its calculation but I need this
>> >> confirmed in order to create a proper patch.
>> >> --
>> >> View this message in context:
>> >>
>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27910673.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source Integration
>> > http://fusesource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27950855.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
>
--
View this message in context: http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27987505.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: prefetchExtension off-by-1 for transacted consumers with
prefetchSize > 0
Posted by Gary Tully <ga...@gmail.com>.
Fair point. From the perspective of the ResourceAdapter where there are
explicit expectations this makes good sense. The prefetch should be written
in stone.
>From a regular transacted consumers perspective, where the prefetch is
largely hidden, avoiding a hang for large transactions is good because a
hang would be hard to diagnose. I agree it could be considered user/error
but it is difficult to flag it as such.
It may make sense to make the (extend the prefetch for a
transaction) behavior configurable such that it can be disabled for the RA
or off by default and enabled in the event that large transactions hang.
Thinking more, off by default is probably the best approach as transactions
that exceed the prefetch should be rare and the RAR behavior would match
expectations.
On 18 March 2010 22:05, rodos77 <eu...@nexj.com> wrote:
>
> I don't really get this. If you extend the prefetchSize to more than what
> it
> was configured to be by the maxMessages parameter in the
> Connection.createConnectionConsumer() call, you are violating the JMS spec.
> The maxMessages parameter is defined as "the maximum number of messages
> that
> can be assigned to a server session at one time", so how can you make it
> bigger than what it was set to by the ConnectionConsumer?!
>
> ActiveMQ Resource Adapter has 2 parameters in its ActivationSpec that
> control the maxMessages argument used in the
> Connection.createConnectionConsumer() call: maxSessions and
> maxMessagesPerSessions. The maxMessages argument is set to their product.
> This makes sense. If a user configures ActiveMQ to have x maxSessions and
> y
> maxMessagesPerSessions, maxMessages (and consequentially the prefetchSize)
> should be set to x*y. But why should this ever be extended? Shouldn't it
> be expected that only a maximum of x*y messages be delivered at the same
> time? If a user wants to process z <= x*y messages in the same tx, that
> should not pose any problem as the RA will have x ServerSessions each
> capable of processing y messages at the same time. But if the user wants
> to
> process z > x*y messages at the same time after explicitly configuring
> ActiveMQ to only deliver a maximum of x*y messages at the same time, this
> is
> a user error/problem. ActiveMQ should not be violating the JMS Spec to
> accommodate this. What is the point of having those properties in the 1st
> place if you are going to extend them and deliver as many messages as
> available?
>
>
>
> Gary Tully wrote:
> >
> > The prefetch extension in the transacted case is necessary for the case
> > where a transacted consumer wants to consumer more than prefetch messages
> > in
> > a single transaction. Without the extension, the broker would see a full
> > prefetch and not dispatch any more and a receive could block.
> >
> > On 15 March 2010 21:31, rodos77 <eu...@nexj.com> wrote:
> >
> >>
> >> JIRA AMQ-2651 opened.
> >>
> >> I would be happy to provide a patch but as I stated in my original post,
> >> I
> >> don't fully understand the need for prefetchExtension in the case of a
> >> transacted, asynchronous (prefetchSize > 0) consumer. If somebody could
> >> explain it, that would perhaps give me the required knowledge to come up
> >> with a patch.
> >>
> >> The way I see it now, I would get rid of the prefetchExtension (in the
> >> above
> >> mentioned case) all together, but again I'm probably missing some piece
> >> of
> >> information and I don't want to cause any side effects. Barring that, I
> >> think there is an off-by-1 error in its calculation but I need this
> >> confirmed in order to create a proper patch.
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27910673.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source Integration
> > http://fusesource.com
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/prefetchExtension-off-by-1-for-transacted-consumers-with-prefetchSize-%3E-0-tp27866123p27950855.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com