You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Trevor Pounds (JIRA)" <ji...@apache.org> on 2009/02/10 21:30:00 UTC
[jira] Created: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistence store
Memory leak when marshaling ActiveMQTextMessage to persistence store
--------------------------------------------------------------------
Key: AMQ-2103
URL: https://issues.apache.org/activemq/browse/AMQ-2103
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.2.0
Reporter: Trevor Pounds
When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistent store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=49354#action_49354 ]
Trevor Pounds commented on AMQ-2103:
------------------------------------
Rob did you look at all to see if this affects other message types (i.e. ActiveMQObjectMessage, ActiveMQMapMessage)? I believe some of these these suffer from the same problem.
> Memory leak when marshaling ActiveMQTextMessage to persistent store
> -------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Assignee: Rob Davies
> Fix For: 5.3.0
>
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistence store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Pounds updated AMQ-2103:
-------------------------------
Attachment: jhat_ByteSequence_data_0xe837adb0.htm
attaching jhat heap analysis showing ActiveMQTextMessage->ByteSequence->byte array object holding the message data.
> Memory leak when marshaling ActiveMQTextMessage to persistence store
> --------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistent store
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Tully resolved AMQ-2103.
-----------------------------
Resolution: Fixed
resolving.
addressing any further limitations to the memory usage tracking as it pertains to actual jvm usage will be punted to 6.0
> Memory leak when marshaling ActiveMQTextMessage to persistent store
> -------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Assignee: Gary Tully
> Fix For: 5.5.0
>
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistence store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Pounds updated AMQ-2103:
-------------------------------
Affects Version/s: (was: 5.2.0)
5.0.0
> Memory leak when marshaling ActiveMQTextMessage to persistence store
> --------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistent store
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62520#action_62520 ]
Gary Tully commented on AMQ-2103:
---------------------------------
I think there is merit in the original use case, but soft references would not cut it as the messages are retained in memory by the cursors and there is duplication. The key issue with any destruction of a message in memory is that it requires a sync point. A general MarshallAware interface hook would require a sync on message that would have a large impact on scaleability of topics.
I agree, that there needs to be some tie with available vm memory and cursor memory usage, but that sort of scheme would require major surgery atm.
Memory management is something that is central to the amq 6.0 (apollo) architecture.
> Memory leak when marshaling ActiveMQTextMessage to persistent store
> -------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Assignee: Gary Tully
> Fix For: 5.5.0
>
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistence store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Pounds updated AMQ-2103:
-------------------------------
Environment: ActiveMQ 5.0.0.20-fuse
> Memory leak when marshaling ActiveMQTextMessage to persistence store
> --------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistence store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=49312#action_49312 ]
Trevor Pounds commented on AMQ-2103:
------------------------------------
I also forgot to mention that this probably only affects use cases with something like vm transport where the message stays internal to the same JVM and is not marshaled across the wire.
> Memory leak when marshaling ActiveMQTextMessage to persistence store
> --------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistent store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Pounds reopened AMQ-2103:
--------------------------------
> Memory leak when marshaling ActiveMQTextMessage to persistent store
> -------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Assignee: Gary Tully
> Fix For: 5.5.0
>
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistent store
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Tully reopened AMQ-2103:
-----------------------------
Assignee: Gary Tully (was: Rob Davies)
This fix breaks the topic case with multiple consumers and a vm transport producer. There is contention over marshaling so whacking the marshaled state is not possible, results in missing content for some of the consumers. https://issues.apache.org/activemq/browse/AMQ-2966
Think the fix is to have an reduceMemoryFootprint policy option that when enabled for queues, will clear the marshaled state (can work for map and object messages also) when the message is first persisted. This is a natural sync point that is contention free provided the concurrentStoreAndDispatchQueue KahaDB option that tries to optimize out persistence, is not enabled.
> Memory leak when marshaling ActiveMQTextMessage to persistent store
> -------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Assignee: Gary Tully
> Fix For: 5.3.0
>
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistence store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Pounds updated AMQ-2103:
-------------------------------
Attachment: heap_100_1MB_messages.png
attaching heap usage graph comparing memory before and after applying patch
> Memory leak when marshaling ActiveMQTextMessage to persistence store
> --------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistence store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Pounds updated AMQ-2103:
-------------------------------
Attachment: AMQ-2103.diff
attaching patch that clears ActiveMQTextMessage 'text' field data when marshaling.
> Memory leak when marshaling ActiveMQTextMessage to persistence store
> --------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistence store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Pounds updated AMQ-2103:
-------------------------------
Attachment: Duplicate Message Data (Internal Marshalling).png
attaching memory usage diagram showing how the message data can be duplicated.
> Memory leak when marshaling ActiveMQTextMessage to persistence store
> --------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Attachments: Duplicate Message Data (Internal Marshalling).png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistent store
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Tully resolved AMQ-2103.
-----------------------------
Resolution: Fixed
Fix Version/s: (was: 5.3.0)
5.5.0
patch reverted and new policy based solution applied in r1021466
The patch, setting the text attribute to null, was unprotected. Currently there are no destructive operations, on a message, allowing it to be shared across topic consumers without synchronization, which allows concurrent dispatch to be fast. This is important to maintain.
A new destination policy for queues, boolean attribute reduceMemoryFootprint has been introduced to do meet this use case.
When true, the marshaled state of a message will be cleared once the message is persisted in the store. This is a natural, contention free sync point.
Note: The kahaDB concurrentStoreAndForwardQueues option (which introduces concurrency) and the memory persistence adapter (which will not marshall) are not compatible with this new policy.
Trevor, can you validate that this change meet your use case?
> Memory leak when marshaling ActiveMQTextMessage to persistent store
> -------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Assignee: Gary Tully
> Fix For: 5.5.0
>
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistent store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Pounds updated AMQ-2103:
-------------------------------
Summary: Memory leak when marshaling ActiveMQTextMessage to persistent store (was: Memory leak when marshaling ActiveMQTextMessage to persistence store)
> Memory leak when marshaling ActiveMQTextMessage to persistent store
> -------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistent store
Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rob Davies resolved AMQ-2103.
-----------------------------
Resolution: Fixed
Fix Version/s: 5.3.0
Patch applied in SVN revision 743258
> Memory leak when marshaling ActiveMQTextMessage to persistent store
> -------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Assignee: Rob Davies
> Fix For: 5.3.0
>
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistence store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Pounds updated AMQ-2103:
-------------------------------
Attachment: jhat_ByteSequence_0xe837a5c0.htm
attaching jhat heap analysis showing ActiveMQTextMessage->ByteSequence object holding the byte array data.
> Memory leak when marshaling ActiveMQTextMessage to persistence store
> --------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Attachments: Duplicate Message Data (Internal Marshalling).png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistent store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62505#action_62505 ]
Trevor Pounds commented on AMQ-2103:
------------------------------------
This fix looks like it will work but I do have some concerns implementing it as a destination policy. Wouldn't it make more sense to implement this using SoftReferences or a hook into the MarshallAware interface? In my opinion it makes sense to clear memory at the discretion of the collector when the message has been completely serialized to disk when there is not enough memory. The actual problem I was trying to solve was to prevent "hidden" memory from hanging around that was not tied to the ActiveMQ memory usage policies. I could be missing something here but it seems there are disjoint philosophies for memory management related to overall broker memory usage, fast dispatch cursors and other unaccounted for "live" objects. I'm not sure what to do here but the proposed fix (including my original submission) seems like a kludge at best. Does anyone else have other opinions?
> Memory leak when marshaling ActiveMQTextMessage to persistent store
> -------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Assignee: Gary Tully
> Fix For: 5.5.0
>
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistence store
Posted by "Trevor Pounds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Trevor Pounds updated AMQ-2103:
-------------------------------
Attachment: jhat_ActiveMQTextMessage_0xe837a478.htm
attaching jhat heap analysis showing ActiveMQTextMessage object with duplicate data.
> Memory leak when marshaling ActiveMQTextMessage to persistence store
> --------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Attachments: Duplicate Message Data (Internal Marshalling).png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AMQ-2103) Memory leak when marshaling
ActiveMQTextMessage to persistent store
Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rob Davies reassigned AMQ-2103:
-------------------------------
Assignee: Rob Davies
> Memory leak when marshaling ActiveMQTextMessage to persistent store
> -------------------------------------------------------------------
>
> Key: AMQ-2103
> URL: https://issues.apache.org/activemq/browse/AMQ-2103
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.0.0
> Environment: ActiveMQ 5.0.0.20-fuse
> Reporter: Trevor Pounds
> Assignee: Rob Davies
> Attachments: AMQ-2103.diff, Duplicate Message Data (Internal Marshalling).png, heap_100_1MB_messages.png, jhat_ActiveMQTextMessage_0xe837a478.htm, jhat_ByteSequence_0xe837a5c0.htm, jhat_ByteSequence_data_0xe837adb0.htm
>
>
> When an org.apache.activemq.command.ActiveMQTextMessage is marshaled into the persistence store some portion of the messages are stored in memory (i.e. pending cursor/consumer dispatch queue). The messages stored in memory have the potential to cause the broker to run out of memory because org.apache.activemq.command.ActiveMQTextMessage objects can store the data twice, once in the 'text' field and once in the 'content' field. Normally this isn't a problem since the 'content' field is cleared when the message is being used in a client application (i.e. by calling getText() clears content). The problem occurs when a consumer is slow and a large number of messages are sitting around on the broker in pending/dispatch memory space. The message is marshaled for the store and then persisted to disk and copied to pending memory when space is available.
> This bug affects any ActiveMQ*Message object that does not clear its temporary data (i.e. 'text' field) once it has been marshaled. When a message is marshaled we should null the derived objects memory space once the data has been written to the parent object's 'content' field.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.