You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "John Heitmann (JIRA)" <ji...@apache.org> on 2006/09/21 09:21:23 UTC
[jira] Created: (AMQ-932) Quickly broken client connections lead to
memory leaks
Quickly broken client connections lead to memory leaks
------------------------------------------------------
Key: AMQ-932
URL: https://issues.apache.org/activemq/browse/AMQ-932
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 4.0.2
Reporter: John Heitmann
Attachments: 73.diff
Connections to the openwire port that are pathologically broken (for example any http request) or that die in some other way extremely quickly will lead to memory losses of aout 64Kb each time. This happens because many services are stop()ed directly in the middle of start(), and then never stopped for real, or stopped again but on an object tree with an inconsistent state. This is usually also accompanied by the JMX message:
WARN ManagedTransportConnection - Failed to unregister mbean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=default,Connection=25
But that is a cosmetic symptom and not critical (and this has otherwise nothing to do with JMX).
My patch is a band-aid that is functional but I'm not very happy with it. The patch changes some service logic so that if stop is called in the middle of start, the stop is instead queued and called at the end of start. There will still be multiple stops, and you'll still see the cosmetic JMX error from the second ineffectual stop, but the first stop cleans up correctly so there are no leaks.
I think there's probably a better solution, but it was tough to see what. I'd appreciate better ideas. Possibly something involving moving the dangerous operations (wire format negotiation etc) out of start?
I am working on a unit test, but I can't promise I will have something to submit. I'm having to play JVM games to detect the problem in a unit test and that might not fly for general purpose use.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-932) Quickly broken client connections lead
to memory leaks
Posted by "John Heitmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-932?page=comments#action_37000 ]
John Heitmann commented on AMQ-932:
-----------------------------------
Those are the exact symptoms that occur as a result of the problem I saw. For every bad connection there was a bunch of individual object leakage (one each of the major transport filters, a couple different mbeans, a connection object etc), but by far the most serious leakage were the empty pre-allocated marshaler cache arrays. This looks like a common symptom of anything that would leak transports/connections, so it might not be the same root cause.
If you can reproduce it and want help I'd be happy to take a look.
If you can't find the problem one thing that could really help mitigate the severity of these kinds of leaks is to make the initialization of the marshaling caches conditional on the wireFormat.cacheEnabled flag. This would give a lot more leak headroom.
> Quickly broken client connections lead to memory leaks
> ------------------------------------------------------
>
> Key: AMQ-932
> URL: https://issues.apache.org/activemq/browse/AMQ-932
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 4.0.2
> Reporter: John Heitmann
> Attachments: 73.diff
>
>
> Connections to the openwire port that are pathologically broken (for example any http request) or that die in some other way extremely quickly will lead to memory losses of aout 64Kb each time. This happens because many services are stop()ed directly in the middle of start(), and then never stopped for real, or stopped again but on an object tree with an inconsistent state. This is usually also accompanied by the JMX message:
> WARN ManagedTransportConnection - Failed to unregister mbean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=default,Connection=25
> But that is a cosmetic symptom and not critical (and this has otherwise nothing to do with JMX).
> My patch is a band-aid that is functional but I'm not very happy with it. The patch changes some service logic so that if stop is called in the middle of start, the stop is instead queued and called at the end of start. There will still be multiple stops, and you'll still see the cosmetic JMX error from the second ineffectual stop, but the first stop cleans up correctly so there are no leaks.
> I think there's probably a better solution, but it was tough to see what. I'd appreciate better ideas. Possibly something involving moving the dangerous operations (wire format negotiation etc) out of start?
> I am working on a unit test, but I can't promise I will have something to submit. I'm having to play JVM games to detect the problem in a unit test and that might not fly for general purpose use.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQ-932) Quickly broken client connections lead
to memory leaks
Posted by "Kelly Campbell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-932?page=comments#action_36997 ]
Kelly Campbell commented on AMQ-932:
------------------------------------
I ran into a similar issue but I don't think it was due to broken connections. I've got a heap dump from the broker (jdk1.6), and it looks like the OpenWireFormat marshalCache is the root cause of the one I'm working on. Each DataStructure[] marshalCache is roughly 64K because of the MARSHAL_CACHE_SIZE. There's 738 DataStructure[] arrays in my heap, taking up 48 Mbytes. Most of these only have one element used.
I think issues 914 and 937 might be related or the same.
> Quickly broken client connections lead to memory leaks
> ------------------------------------------------------
>
> Key: AMQ-932
> URL: https://issues.apache.org/activemq/browse/AMQ-932
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 4.0.2
> Reporter: John Heitmann
> Attachments: 73.diff
>
>
> Connections to the openwire port that are pathologically broken (for example any http request) or that die in some other way extremely quickly will lead to memory losses of aout 64Kb each time. This happens because many services are stop()ed directly in the middle of start(), and then never stopped for real, or stopped again but on an object tree with an inconsistent state. This is usually also accompanied by the JMX message:
> WARN ManagedTransportConnection - Failed to unregister mbean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=default,Connection=25
> But that is a cosmetic symptom and not critical (and this has otherwise nothing to do with JMX).
> My patch is a band-aid that is functional but I'm not very happy with it. The patch changes some service logic so that if stop is called in the middle of start, the stop is instead queued and called at the end of start. There will still be multiple stops, and you'll still see the cosmetic JMX error from the second ineffectual stop, but the first stop cleans up correctly so there are no leaks.
> I think there's probably a better solution, but it was tough to see what. I'd appreciate better ideas. Possibly something involving moving the dangerous operations (wire format negotiation etc) out of start?
> I am working on a unit test, but I can't promise I will have something to submit. I'm having to play JVM games to detect the problem in a unit test and that might not fly for general purpose use.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (AMQ-932) Quickly broken client connections lead
to memory leaks
Posted by "james strachan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-932?page=all ]
james strachan resolved AMQ-932.
--------------------------------
Fix Version/s: 4.1
Resolution: Fixed
Patch applied John with thanks - could you double check I've applied it correctly please?
> Quickly broken client connections lead to memory leaks
> ------------------------------------------------------
>
> Key: AMQ-932
> URL: https://issues.apache.org/activemq/browse/AMQ-932
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 4.0.2
> Reporter: John Heitmann
> Fix For: 4.1
>
> Attachments: 73.diff
>
>
> Connections to the openwire port that are pathologically broken (for example any http request) or that die in some other way extremely quickly will lead to memory losses of aout 64Kb each time. This happens because many services are stop()ed directly in the middle of start(), and then never stopped for real, or stopped again but on an object tree with an inconsistent state. This is usually also accompanied by the JMX message:
> WARN ManagedTransportConnection - Failed to unregister mbean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=default,Connection=25
> But that is a cosmetic symptom and not critical (and this has otherwise nothing to do with JMX).
> My patch is a band-aid that is functional but I'm not very happy with it. The patch changes some service logic so that if stop is called in the middle of start, the stop is instead queued and called at the end of start. There will still be multiple stops, and you'll still see the cosmetic JMX error from the second ineffectual stop, but the first stop cleans up correctly so there are no leaks.
> I think there's probably a better solution, but it was tough to see what. I'd appreciate better ideas. Possibly something involving moving the dangerous operations (wire format negotiation etc) out of start?
> I am working on a unit test, but I can't promise I will have something to submit. I'm having to play JVM games to detect the problem in a unit test and that might not fly for general purpose use.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira