You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Fabrice Delaporte (JIRA)" <ji...@apache.org> on 2009/12/14 15:41:52 UTC
[jira] Created: (AMQ-2532) Broker paging all available messages
from store leading to resource exhaustion
Broker paging all available messages from store leading to resource exhaustion
------------------------------------------------------------------------------
Key: AMQ-2532
URL: https://issues.apache.org/activemq/browse/AMQ-2532
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.3.0
Environment: Java 1.6
Reporter: Fabrice Delaporte
There seems to be a regression (works fine in 5.2.0) related to the fix for the following:
https://issues.apache.org/activemq/browse/AMQ-2413
I think the intent of the original code was to prevent uncontrolled paging of pending messages. The doPageIn method is called in version 5.3.0 :
- each time a message is acked/committed
- each time the MemoryUsage is decreased
As a result, if you create a queue with, say, 1 000 000 messages in it, then start a consumer, the broker will try to page in all the messages from the store as consuming goes on (with default configuration values, 200 paged in messages for one consumed message minimum - I say minimum because if the MemoryUsage is decreased when you consume a message, then doPageIn is invoked again). Guaranteed OutOfMemoryError as in general consuming messages is slower than paging from store. I guess the same can occur when you have a fast producer and a slow consumer, without producer flow control.
I would say that in any case, the broker should never page in more messages than the maxPageSize attribute, plus the number of dispatched messages. More formally:
{code:java}
int toPageIn = getMaxPageSize() - pagedInPendingDispatch.size();
{code}
And just to make sure we never get a negative value:
{code:java}
int toPageIn = Max(0, getMaxPageSize() - pagedInPendingDispatch.size());
{code}
Note 1: this behavior is also exhibited by the background message expiration logic, that forces message pagination each time it is run. With big queues without consumers, by default you'll get 200 more messages paginated each time the expiration task is run. Eventually leading to broker resource exhaustion if no consumer comes into play in the near future with active producers queuing up messages.
Note 2: this may also be related to https://issues.apache.org/activemq/browse/AMQ-2468
Workarounds:
- setting lazyDispatch to true fixes the problem for message consuming only
- background expiration can be disabled by setting expireMessagesPeriod to -1
Cheers,
Fabrice
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2532) Broker paging all available messages
from store leading to resource exhaustion
Posted by "Fabrice Delaporte (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=56435#action_56435 ]
Fabrice Delaporte commented on AMQ-2532:
----------------------------------------
Ah, yes indeed, this is fixed. Good news !
Strange it didn't strike me that I should try the latest snapshots before reporting ;)
Sorry for the noise.
Cheers,
Fabrice
> Broker paging all available messages from store leading to resource exhaustion
> ------------------------------------------------------------------------------
>
> Key: AMQ-2532
> URL: https://issues.apache.org/activemq/browse/AMQ-2532
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.0
> Environment: Java 1.6
> Reporter: Fabrice Delaporte
>
> There seems to be a regression (works fine in 5.2.0) related to the fix for the following:
> https://issues.apache.org/activemq/browse/AMQ-2413
> I think the intent of the original code was to prevent uncontrolled paging of pending messages. The doPageIn method is called in version 5.3.0 :
> - each time a message is acked/committed
> - each time the MemoryUsage is decreased
> As a result, if you create a queue with, say, 1 000 000 messages in it, then start a consumer, the broker will try to page in all the messages from the store as consuming goes on (with default configuration values, 200 paged in messages for one consumed message minimum - I say minimum because if the MemoryUsage is decreased when you consume a message, then doPageIn is invoked again). Guaranteed OutOfMemoryError as in general consuming messages is slower than paging from store. I guess the same can occur when you have a fast producer and a slow consumer, without producer flow control.
> I would say that in any case, the broker should never page in more messages than the maxPageSize attribute, plus the number of dispatched messages. More formally:
> {code:java}
> int toPageIn = getMaxPageSize() - pagedInPendingDispatch.size();
> {code}
> And just to make sure we never get a negative value:
> {code:java}
> int toPageIn = Math.max(0, getMaxPageSize() - pagedInPendingDispatch.size());
> {code}
> Note 1: this behavior is also exhibited by the background message expiration logic, that forces message pagination each time it is run. With big queues without consumers, by default you'll get 200 more messages paginated each time the expiration task is run. Eventually leading to broker resource exhaustion if no consumer comes into play in the near future with active producers queuing up messages.
> Note 2: this may also be related to https://issues.apache.org/activemq/browse/AMQ-2468
> Workarounds:
> - setting lazyDispatch to true fixes the problem for message consuming only
> - background expiration can be disabled by setting expireMessagesPeriod to -1
> Cheers,
> Fabrice
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2532) Broker paging all available messages
from store leading to resource exhaustion
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=56432#action_56432 ]
Gary Tully commented on AMQ-2532:
---------------------------------
I suspect that this is resolved on trunk via https://issues.apache.org/activemq/browse/AMQ-2468 and https://issues.apache.org/activemq/browse/AMQ-2481
Can you validate with a 5.3.1-SNAPSHOT of 5.4-SNAPSHOT
> Broker paging all available messages from store leading to resource exhaustion
> ------------------------------------------------------------------------------
>
> Key: AMQ-2532
> URL: https://issues.apache.org/activemq/browse/AMQ-2532
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.0
> Environment: Java 1.6
> Reporter: Fabrice Delaporte
>
> There seems to be a regression (works fine in 5.2.0) related to the fix for the following:
> https://issues.apache.org/activemq/browse/AMQ-2413
> I think the intent of the original code was to prevent uncontrolled paging of pending messages. The doPageIn method is called in version 5.3.0 :
> - each time a message is acked/committed
> - each time the MemoryUsage is decreased
> As a result, if you create a queue with, say, 1 000 000 messages in it, then start a consumer, the broker will try to page in all the messages from the store as consuming goes on (with default configuration values, 200 paged in messages for one consumed message minimum - I say minimum because if the MemoryUsage is decreased when you consume a message, then doPageIn is invoked again). Guaranteed OutOfMemoryError as in general consuming messages is slower than paging from store. I guess the same can occur when you have a fast producer and a slow consumer, without producer flow control.
> I would say that in any case, the broker should never page in more messages than the maxPageSize attribute, plus the number of dispatched messages. More formally:
> {code:java}
> int toPageIn = getMaxPageSize() - pagedInPendingDispatch.size();
> {code}
> And just to make sure we never get a negative value:
> {code:java}
> int toPageIn = Math.max(0, getMaxPageSize() - pagedInPendingDispatch.size());
> {code}
> Note 1: this behavior is also exhibited by the background message expiration logic, that forces message pagination each time it is run. With big queues without consumers, by default you'll get 200 more messages paginated each time the expiration task is run. Eventually leading to broker resource exhaustion if no consumer comes into play in the near future with active producers queuing up messages.
> Note 2: this may also be related to https://issues.apache.org/activemq/browse/AMQ-2468
> Workarounds:
> - setting lazyDispatch to true fixes the problem for message consuming only
> - background expiration can be disabled by setting expireMessagesPeriod to -1
> Cheers,
> Fabrice
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (AMQ-2532) Broker paging all available messages from
store leading to resource exhaustion
Posted by "Fabrice Delaporte (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fabrice Delaporte closed AMQ-2532.
----------------------------------
Resolution: Fixed
Fix Version/s: 5.4.0
5.3.1
This is already fixed.
> Broker paging all available messages from store leading to resource exhaustion
> ------------------------------------------------------------------------------
>
> Key: AMQ-2532
> URL: https://issues.apache.org/activemq/browse/AMQ-2532
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.0
> Environment: Java 1.6
> Reporter: Fabrice Delaporte
> Fix For: 5.3.1, 5.4.0
>
>
> There seems to be a regression (works fine in 5.2.0) related to the fix for the following:
> https://issues.apache.org/activemq/browse/AMQ-2413
> I think the intent of the original code was to prevent uncontrolled paging of pending messages. The doPageIn method is called in version 5.3.0 :
> - each time a message is acked/committed
> - each time the MemoryUsage is decreased
> As a result, if you create a queue with, say, 1 000 000 messages in it, then start a consumer, the broker will try to page in all the messages from the store as consuming goes on (with default configuration values, 200 paged in messages for one consumed message minimum - I say minimum because if the MemoryUsage is decreased when you consume a message, then doPageIn is invoked again). Guaranteed OutOfMemoryError as in general consuming messages is slower than paging from store. I guess the same can occur when you have a fast producer and a slow consumer, without producer flow control.
> I would say that in any case, the broker should never page in more messages than the maxPageSize attribute, plus the number of dispatched messages. More formally:
> {code:java}
> int toPageIn = getMaxPageSize() - pagedInPendingDispatch.size();
> {code}
> And just to make sure we never get a negative value:
> {code:java}
> int toPageIn = Math.max(0, getMaxPageSize() - pagedInPendingDispatch.size());
> {code}
> Note 1: this behavior is also exhibited by the background message expiration logic, that forces message pagination each time it is run. With big queues without consumers, by default you'll get 200 more messages paginated each time the expiration task is run. Eventually leading to broker resource exhaustion if no consumer comes into play in the near future with active producers queuing up messages.
> Note 2: this may also be related to https://issues.apache.org/activemq/browse/AMQ-2468
> Workarounds:
> - setting lazyDispatch to true fixes the problem for message consuming only
> - background expiration can be disabled by setting expireMessagesPeriod to -1
> Cheers,
> Fabrice
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2532) Broker paging all available messages
from store leading to resource exhaustion
Posted by "Fabrice Delaporte (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fabrice Delaporte updated AMQ-2532:
-----------------------------------
Description:
There seems to be a regression (works fine in 5.2.0) related to the fix for the following:
https://issues.apache.org/activemq/browse/AMQ-2413
I think the intent of the original code was to prevent uncontrolled paging of pending messages. The doPageIn method is called in version 5.3.0 :
- each time a message is acked/committed
- each time the MemoryUsage is decreased
As a result, if you create a queue with, say, 1 000 000 messages in it, then start a consumer, the broker will try to page in all the messages from the store as consuming goes on (with default configuration values, 200 paged in messages for one consumed message minimum - I say minimum because if the MemoryUsage is decreased when you consume a message, then doPageIn is invoked again). Guaranteed OutOfMemoryError as in general consuming messages is slower than paging from store. I guess the same can occur when you have a fast producer and a slow consumer, without producer flow control.
I would say that in any case, the broker should never page in more messages than the maxPageSize attribute, plus the number of dispatched messages. More formally:
{code:java}
int toPageIn = getMaxPageSize() - pagedInPendingDispatch.size();
{code}
And just to make sure we never get a negative value:
{code:java}
int toPageIn = Math.max(0, getMaxPageSize() - pagedInPendingDispatch.size());
{code}
Note 1: this behavior is also exhibited by the background message expiration logic, that forces message pagination each time it is run. With big queues without consumers, by default you'll get 200 more messages paginated each time the expiration task is run. Eventually leading to broker resource exhaustion if no consumer comes into play in the near future with active producers queuing up messages.
Note 2: this may also be related to https://issues.apache.org/activemq/browse/AMQ-2468
Workarounds:
- setting lazyDispatch to true fixes the problem for message consuming only
- background expiration can be disabled by setting expireMessagesPeriod to -1
Cheers,
Fabrice
was:
There seems to be a regression (works fine in 5.2.0) related to the fix for the following:
https://issues.apache.org/activemq/browse/AMQ-2413
I think the intent of the original code was to prevent uncontrolled paging of pending messages. The doPageIn method is called in version 5.3.0 :
- each time a message is acked/committed
- each time the MemoryUsage is decreased
As a result, if you create a queue with, say, 1 000 000 messages in it, then start a consumer, the broker will try to page in all the messages from the store as consuming goes on (with default configuration values, 200 paged in messages for one consumed message minimum - I say minimum because if the MemoryUsage is decreased when you consume a message, then doPageIn is invoked again). Guaranteed OutOfMemoryError as in general consuming messages is slower than paging from store. I guess the same can occur when you have a fast producer and a slow consumer, without producer flow control.
I would say that in any case, the broker should never page in more messages than the maxPageSize attribute, plus the number of dispatched messages. More formally:
{code:java}
int toPageIn = getMaxPageSize() - pagedInPendingDispatch.size();
{code}
And just to make sure we never get a negative value:
{code:java}
int toPageIn = Max(0, getMaxPageSize() - pagedInPendingDispatch.size());
{code}
Note 1: this behavior is also exhibited by the background message expiration logic, that forces message pagination each time it is run. With big queues without consumers, by default you'll get 200 more messages paginated each time the expiration task is run. Eventually leading to broker resource exhaustion if no consumer comes into play in the near future with active producers queuing up messages.
Note 2: this may also be related to https://issues.apache.org/activemq/browse/AMQ-2468
Workarounds:
- setting lazyDispatch to true fixes the problem for message consuming only
- background expiration can be disabled by setting expireMessagesPeriod to -1
Cheers,
Fabrice
> Broker paging all available messages from store leading to resource exhaustion
> ------------------------------------------------------------------------------
>
> Key: AMQ-2532
> URL: https://issues.apache.org/activemq/browse/AMQ-2532
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.0
> Environment: Java 1.6
> Reporter: Fabrice Delaporte
>
> There seems to be a regression (works fine in 5.2.0) related to the fix for the following:
> https://issues.apache.org/activemq/browse/AMQ-2413
> I think the intent of the original code was to prevent uncontrolled paging of pending messages. The doPageIn method is called in version 5.3.0 :
> - each time a message is acked/committed
> - each time the MemoryUsage is decreased
> As a result, if you create a queue with, say, 1 000 000 messages in it, then start a consumer, the broker will try to page in all the messages from the store as consuming goes on (with default configuration values, 200 paged in messages for one consumed message minimum - I say minimum because if the MemoryUsage is decreased when you consume a message, then doPageIn is invoked again). Guaranteed OutOfMemoryError as in general consuming messages is slower than paging from store. I guess the same can occur when you have a fast producer and a slow consumer, without producer flow control.
> I would say that in any case, the broker should never page in more messages than the maxPageSize attribute, plus the number of dispatched messages. More formally:
> {code:java}
> int toPageIn = getMaxPageSize() - pagedInPendingDispatch.size();
> {code}
> And just to make sure we never get a negative value:
> {code:java}
> int toPageIn = Math.max(0, getMaxPageSize() - pagedInPendingDispatch.size());
> {code}
> Note 1: this behavior is also exhibited by the background message expiration logic, that forces message pagination each time it is run. With big queues without consumers, by default you'll get 200 more messages paginated each time the expiration task is run. Eventually leading to broker resource exhaustion if no consumer comes into play in the near future with active producers queuing up messages.
> Note 2: this may also be related to https://issues.apache.org/activemq/browse/AMQ-2468
> Workarounds:
> - setting lazyDispatch to true fixes the problem for message consuming only
> - background expiration can be disabled by setting expireMessagesPeriod to -1
> Cheers,
> Fabrice
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.