You are viewing a plain text version of this content. The canonical link for it is here.
Posted to rampart-dev@ws.apache.org by "Stefano Bruna (JIRA)" <ji...@apache.org> on 2010/06/22 16:05:58 UTC
[jira] Created: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
------------------------------------------------------------------------------------------
Key: AXIS2-4749
URL: https://issues.apache.org/jira/browse/AXIS2-4749
Project: Axis2
Issue Type: Bug
Affects Versions: 1.6
Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
Reporter: Stefano Bruna
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Updated: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefano Bruna updated AXIS2-4749:
---------------------------------
Description:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry; <===== entry with the same hash
key = obj; <======= orginal object, needed for the equals use in case of same hash
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
was:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Reopened: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez reopened AXIS2-4749:
---------------------------------
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Issue Comment Edited: (AXIS2-4749) Tribe's
AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big
messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881270#action_12881270 ]
Afkham Azeez edited comment on AXIS2-4749 at 6/22/10 2:02 PM:
--------------------------------------------------------------
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using a configuration parameter in the axis2.xml file.
was (Author: afkham_azeez):
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using an configuration parameter in the axis2.xml file.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Updated: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefano Bruna updated AXIS2-4749:
---------------------------------
Description:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
was:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry;
> key = obj; <======= orginal object !!!
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
Re: [jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by Glen Daniels <gl...@thoughtcraft.com>.
Fast work, Azeez! Nice. Only comment - shouldn't the resolution be "fixed"
instead of "invalid"?
--G
On 6/22/2010 2:55 PM, Afkham Azeez (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Afkham Azeez resolved AXIS2-4749.
> ---------------------------------
>
> Assignee: Afkham Azeez
> Resolution: Invalid
>
> Fixed in the trunk. Revision: 956974
>
>> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
>> ------------------------------------------------------------------------------------------
>>
>> Key: AXIS2-4749
>> URL: https://issues.apache.org/jira/browse/AXIS2-4749
>> Project: Axis2
>> Issue Type: Bug
>> Affects Versions: 1.6
>> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
>> Reporter: Stefano Bruna
>> Assignee: Afkham Azeez
>>
>> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
>> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
>> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
>> Somewhere inside HashMap.java
>> // private variables
>> final Object key;
>> Object value;
>> Entry next;
>> final int hash;
>> // constructor
>> Entry(int i, Object obj, Object obj1, Entry entry)
>> {
>> value = obj1; <==== value, ok
>> next = entry; <===== entry with the same hash
>> key = obj; <======= orginal object, needed for the equals use in case of same hash
>> hash = i; <======== key, ok
>> }
>> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
>> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
>> .....
>> // map
>> private static final Map<Integer, Long> receivedMessagesHashCodes =
>> new HashMap<Integer, Long>();
>> .....
>> Integer hashCode = new Integer(msg.hashCode());
>>
>> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
>> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
>> super.messageReceived(msg);
>> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
>> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
>> }
>> etc...
>> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
Re: [jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by Glen Daniels <gl...@thoughtcraft.com>.
Fast work, Azeez! Nice. Only comment - shouldn't the resolution be "fixed"
instead of "invalid"?
--G
On 6/22/2010 2:55 PM, Afkham Azeez (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Afkham Azeez resolved AXIS2-4749.
> ---------------------------------
>
> Assignee: Afkham Azeez
> Resolution: Invalid
>
> Fixed in the trunk. Revision: 956974
>
>> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
>> ------------------------------------------------------------------------------------------
>>
>> Key: AXIS2-4749
>> URL: https://issues.apache.org/jira/browse/AXIS2-4749
>> Project: Axis2
>> Issue Type: Bug
>> Affects Versions: 1.6
>> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
>> Reporter: Stefano Bruna
>> Assignee: Afkham Azeez
>>
>> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
>> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
>> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
>> Somewhere inside HashMap.java
>> // private variables
>> final Object key;
>> Object value;
>> Entry next;
>> final int hash;
>> // constructor
>> Entry(int i, Object obj, Object obj1, Entry entry)
>> {
>> value = obj1; <==== value, ok
>> next = entry; <===== entry with the same hash
>> key = obj; <======= orginal object, needed for the equals use in case of same hash
>> hash = i; <======== key, ok
>> }
>> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
>> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
>> .....
>> // map
>> private static final Map<Integer, Long> receivedMessagesHashCodes =
>> new HashMap<Integer, Long>();
>> .....
>> Integer hashCode = new Integer(msg.hashCode());
>>
>> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
>> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
>> super.messageReceived(msg);
>> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
>> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
>> }
>> etc...
>> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
Re: [jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by Glen Daniels <gl...@thoughtcraft.com>.
Fast work, Azeez! Nice. Only comment - shouldn't the resolution be "fixed"
instead of "invalid"?
--G
On 6/22/2010 2:55 PM, Afkham Azeez (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Afkham Azeez resolved AXIS2-4749.
> ---------------------------------
>
> Assignee: Afkham Azeez
> Resolution: Invalid
>
> Fixed in the trunk. Revision: 956974
>
>> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
>> ------------------------------------------------------------------------------------------
>>
>> Key: AXIS2-4749
>> URL: https://issues.apache.org/jira/browse/AXIS2-4749
>> Project: Axis2
>> Issue Type: Bug
>> Affects Versions: 1.6
>> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
>> Reporter: Stefano Bruna
>> Assignee: Afkham Azeez
>>
>> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
>> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
>> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
>> Somewhere inside HashMap.java
>> // private variables
>> final Object key;
>> Object value;
>> Entry next;
>> final int hash;
>> // constructor
>> Entry(int i, Object obj, Object obj1, Entry entry)
>> {
>> value = obj1; <==== value, ok
>> next = entry; <===== entry with the same hash
>> key = obj; <======= orginal object, needed for the equals use in case of same hash
>> hash = i; <======== key, ok
>> }
>> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
>> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
>> .....
>> // map
>> private static final Map<Integer, Long> receivedMessagesHashCodes =
>> new HashMap<Integer, Long>();
>> .....
>> Integer hashCode = new Integer(msg.hashCode());
>>
>> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
>> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
>> super.messageReceived(msg);
>> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
>> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
>> }
>> etc...
>> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
Re: [jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by Glen Daniels <gl...@thoughtcraft.com>.
Fast work, Azeez! Nice. Only comment - shouldn't the resolution be "fixed"
instead of "invalid"?
--G
On 6/22/2010 2:55 PM, Afkham Azeez (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Afkham Azeez resolved AXIS2-4749.
> ---------------------------------
>
> Assignee: Afkham Azeez
> Resolution: Invalid
>
> Fixed in the trunk. Revision: 956974
>
>> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
>> ------------------------------------------------------------------------------------------
>>
>> Key: AXIS2-4749
>> URL: https://issues.apache.org/jira/browse/AXIS2-4749
>> Project: Axis2
>> Issue Type: Bug
>> Affects Versions: 1.6
>> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
>> Reporter: Stefano Bruna
>> Assignee: Afkham Azeez
>>
>> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
>> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
>> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
>> Somewhere inside HashMap.java
>> // private variables
>> final Object key;
>> Object value;
>> Entry next;
>> final int hash;
>> // constructor
>> Entry(int i, Object obj, Object obj1, Entry entry)
>> {
>> value = obj1; <==== value, ok
>> next = entry; <===== entry with the same hash
>> key = obj; <======= orginal object, needed for the equals use in case of same hash
>> hash = i; <======== key, ok
>> }
>> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
>> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
>> .....
>> // map
>> private static final Map<Integer, Long> receivedMessagesHashCodes =
>> new HashMap<Integer, Long>();
>> .....
>> Integer hashCode = new Integer(msg.hashCode());
>>
>> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
>> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
>> super.messageReceived(msg);
>> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
>> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
>> }
>> etc...
>> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
Re: [jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by Glen Daniels <gl...@thoughtcraft.com>.
Fast work, Azeez! Nice. Only comment - shouldn't the resolution be "fixed"
instead of "invalid"?
--G
On 6/22/2010 2:55 PM, Afkham Azeez (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>
> Afkham Azeez resolved AXIS2-4749.
> ---------------------------------
>
> Assignee: Afkham Azeez
> Resolution: Invalid
>
> Fixed in the trunk. Revision: 956974
>
>> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
>> ------------------------------------------------------------------------------------------
>>
>> Key: AXIS2-4749
>> URL: https://issues.apache.org/jira/browse/AXIS2-4749
>> Project: Axis2
>> Issue Type: Bug
>> Affects Versions: 1.6
>> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
>> Reporter: Stefano Bruna
>> Assignee: Afkham Azeez
>>
>> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
>> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
>> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
>> Somewhere inside HashMap.java
>> // private variables
>> final Object key;
>> Object value;
>> Entry next;
>> final int hash;
>> // constructor
>> Entry(int i, Object obj, Object obj1, Entry entry)
>> {
>> value = obj1; <==== value, ok
>> next = entry; <===== entry with the same hash
>> key = obj; <======= orginal object, needed for the equals use in case of same hash
>> hash = i; <======== key, ok
>> }
>> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
>> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
>> .....
>> // map
>> private static final Map<Integer, Long> receivedMessagesHashCodes =
>> new HashMap<Integer, Long>();
>> .....
>> Integer hashCode = new Integer(msg.hashCode());
>>
>> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
>> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
>> super.messageReceived(msg);
>> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
>> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
>> }
>> etc...
>> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez resolved AXIS2-4749.
---------------------------------
Assignee: Afkham Azeez
Resolution: Invalid
Fixed in the trunk. Revision: 956974
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Updated: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefano Bruna updated AXIS2-4749:
---------------------------------
Description:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
was:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry;
> key = obj; <======= orginal object !!!
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Reopened: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez reopened AXIS2-4749:
---------------------------------
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Issue Comment Edited: (AXIS2-4749) Tribe's
AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big
messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881270#action_12881270 ]
Afkham Azeez edited comment on AXIS2-4749 at 6/22/10 2:02 PM:
--------------------------------------------------------------
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using a configuration parameter in the axis2.xml file.
was (Author: afkham_azeez):
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using an configuration parameter in the axis2.xml file.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881287#action_12881287 ]
Afkham Azeez commented on AXIS2-4749:
-------------------------------------
We will be able to use the org.apache.catalina.tribes.ChannelMessage#getUniqueId() method to solve this issue. I will make the necessary fixes.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881287#action_12881287 ]
Afkham Azeez commented on AXIS2-4749:
-------------------------------------
We will be able to use the org.apache.catalina.tribes.ChannelMessage#getUniqueId() method to solve this issue. I will make the necessary fixes.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Updated: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefano Bruna updated AXIS2-4749:
---------------------------------
Description:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry; <===== entry with the same hash
key = obj; <======= orginal object, needed for the equals use in case of same hash
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
was:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez resolved AXIS2-4749.
---------------------------------
Assignee: Afkham Azeez
Resolution: Invalid
Fixed in the trunk. Revision: 956974
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Issue Comment Edited: (AXIS2-4749) Tribe's
AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big
messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881270#action_12881270 ]
Afkham Azeez edited comment on AXIS2-4749 at 6/22/10 2:02 PM:
--------------------------------------------------------------
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using a configuration parameter in the axis2.xml file.
was (Author: afkham_azeez):
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using an configuration parameter in the axis2.xml file.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez resolved AXIS2-4749.
---------------------------------
Resolution: Fixed
Oops... my bad. Changing resolution to fixed. Thanks Glen for pointing this out.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Issue Comment Edited: (AXIS2-4749) Tribe's
AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big
messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881292#action_12881292 ]
Stefano Bruna edited comment on AXIS2-4749 at 6/22/10 2:53 PM:
---------------------------------------------------------------
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
UPDATE: ok, i see now you last comment. Thank you.
was (Author: stefanobruna):
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Issue Comment Edited: (AXIS2-4749) Tribe's
AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big
messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881292#action_12881292 ]
Stefano Bruna edited comment on AXIS2-4749 at 6/22/10 2:53 PM:
---------------------------------------------------------------
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
UPDATE: ok, i see now you last comment. Thank you.
was (Author: stefanobruna):
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881292#action_12881292 ]
Stefano Bruna commented on AXIS2-4749:
--------------------------------------
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez resolved AXIS2-4749.
---------------------------------
Resolution: Fixed
Oops... my bad. Changing resolution to fixed. Thanks Glen for pointing this out.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez resolved AXIS2-4749.
---------------------------------
Assignee: Afkham Azeez
Resolution: Invalid
Fixed in the trunk. Revision: 956974
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Reopened: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez reopened AXIS2-4749:
---------------------------------
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez resolved AXIS2-4749.
---------------------------------
Assignee: Afkham Azeez
Resolution: Invalid
Fixed in the trunk. Revision: 956974
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Reopened: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez reopened AXIS2-4749:
---------------------------------
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Issue Comment Edited: (AXIS2-4749) Tribe's
AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big
messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881270#action_12881270 ]
Afkham Azeez edited comment on AXIS2-4749 at 6/22/10 2:02 PM:
--------------------------------------------------------------
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using a configuration parameter in the axis2.xml file.
was (Author: afkham_azeez):
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using an configuration parameter in the axis2.xml file.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881270#action_12881270 ]
Afkham Azeez commented on AXIS2-4749:
-------------------------------------
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using an configuration parameter in the axis2.xml file.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881292#action_12881292 ]
Stefano Bruna commented on AXIS2-4749:
--------------------------------------
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881287#action_12881287 ]
Afkham Azeez commented on AXIS2-4749:
-------------------------------------
We will be able to use the org.apache.catalina.tribes.ChannelMessage#getUniqueId() method to solve this issue. I will make the necessary fixes.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Updated: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefano Bruna updated AXIS2-4749:
---------------------------------
Description:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry; <===== entry with the same hash
key = obj; <======= orginal object, needed for the equals use in case of same hash
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
was:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Reopened: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez reopened AXIS2-4749:
---------------------------------
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881287#action_12881287 ]
Afkham Azeez commented on AXIS2-4749:
-------------------------------------
We will be able to use the org.apache.catalina.tribes.ChannelMessage#getUniqueId() method to solve this issue. I will make the necessary fixes.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881292#action_12881292 ]
Stefano Bruna commented on AXIS2-4749:
--------------------------------------
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez resolved AXIS2-4749.
---------------------------------
Resolution: Fixed
Oops... my bad. Changing resolution to fixed. Thanks Glen for pointing this out.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881292#action_12881292 ]
Stefano Bruna commented on AXIS2-4749:
--------------------------------------
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Updated: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefano Bruna updated AXIS2-4749:
---------------------------------
Description:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
was:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry;
> key = obj; <======= orginal object !!!
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881270#action_12881270 ]
Afkham Azeez commented on AXIS2-4749:
-------------------------------------
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using an configuration parameter in the axis2.xml file.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881270#action_12881270 ]
Afkham Azeez commented on AXIS2-4749:
-------------------------------------
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using an configuration parameter in the axis2.xml file.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Updated: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefano Bruna updated AXIS2-4749:
---------------------------------
Description:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry; <===== entry with the same hash
key = obj; <======= orginal object, needed for the equals use in case of same hash
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
was:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Updated: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefano Bruna updated AXIS2-4749:
---------------------------------
Description:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
was:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry;
> key = obj; <======= orginal object !!!
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Issue Comment Edited: (AXIS2-4749) Tribe's
AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big
messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881292#action_12881292 ]
Stefano Bruna edited comment on AXIS2-4749 at 6/22/10 2:53 PM:
---------------------------------------------------------------
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
UPDATE: ok, i see now you last comment. Thank you.
was (Author: stefanobruna):
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881287#action_12881287 ]
Afkham Azeez commented on AXIS2-4749:
-------------------------------------
We will be able to use the org.apache.catalina.tribes.ChannelMessage#getUniqueId() method to solve this issue. I will make the necessary fixes.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Updated: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefano Bruna updated AXIS2-4749:
---------------------------------
Description:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry; <===== entry with the same hash
key = obj; <======= orginal object, needed for the equals use in case of same hash
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
was:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881270#action_12881270 ]
Afkham Azeez commented on AXIS2-4749:
-------------------------------------
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using an configuration parameter in the axis2.xml file.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Updated: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefano Bruna updated AXIS2-4749:
---------------------------------
Description:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
was:
Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
Somewhere inside HashMap.java
// private variables
final Object key;
Object value;
Entry next;
final int hash;
// constructor
Entry(int i, Object obj, Object obj1, Entry entry)
{
value = obj1; <==== value, ok
next = entry;
key = obj; <======= orginal object !!!
hash = i; <======== key, ok
}
So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
.....
// map
private static final Map<Integer, Long> receivedMessagesHashCodes =
new HashMap<Integer, Long>();
.....
Integer hashCode = new Integer(msg.hashCode());
if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
super.messageReceived(msg);
} else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
}
etc...
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry;
> key = obj; <======= orginal object !!!
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez resolved AXIS2-4749.
---------------------------------
Resolution: Fixed
Oops... my bad. Changing resolution to fixed. Thanks Glen for pointing this out.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Issue Comment Edited: (AXIS2-4749) Tribe's
AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big
messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881270#action_12881270 ]
Afkham Azeez edited comment on AXIS2-4749 at 6/22/10 2:02 PM:
--------------------------------------------------------------
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using a configuration parameter in the axis2.xml file.
was (Author: afkham_azeez):
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using an configuration parameter in the axis2.xml file.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez resolved AXIS2-4749.
---------------------------------
Resolution: Fixed
Oops... my bad. Changing resolution to fixed. Thanks Glen for pointing this out.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Issue Comment Edited: (AXIS2-4749) Tribe's
AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big
messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881292#action_12881292 ]
Stefano Bruna edited comment on AXIS2-4749 at 6/22/10 2:53 PM:
---------------------------------------------------------------
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
UPDATE: ok, i see now you last comment. Thank you.
was (Author: stefanobruna):
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Resolved: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afkham Azeez resolved AXIS2-4749.
---------------------------------
Assignee: Afkham Azeez
Resolution: Invalid
Fixed in the trunk. Revision: 956974
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
> Assignee: Afkham Azeez
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881292#action_12881292 ]
Stefano Bruna commented on AXIS2-4749:
--------------------------------------
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Issue Comment Edited: (AXIS2-4749) Tribe's
AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big
messages.
Posted by "Stefano Bruna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881292#action_12881292 ]
Stefano Bruna edited comment on AXIS2-4749 at 6/22/10 2:53 PM:
---------------------------------------------------------------
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
UPDATE: ok, i see now you last comment. Thank you.
was (Author: stefanobruna):
Yes, I also thought that using only the hash could lead to some false positive duplicate messages as i wrote in the very last line in the issue description.
What we faced was that the 5 minutes task was too slow to clean the memory under heavy load and the VM was reporting us many OutOfMemory.
Is in this situation the entire ESB node is not working until the cleaning thread does its job. This is why we turned off the at-most-once interceptor as you suggest.
In the case the interceptor was needed I was thinking about a FIFO queue base on actule size of the received messages. When you reach a limit, older messages are cleaned even if they are still within the 5 minutes time window.
I tryed to implement it but calculate the actual size of received messages is uselessly complex and slow while here perfomance are really needed.
I'll reduce the 5 minutes timeout and increase server memory :)
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org
[jira] Commented: (AXIS2-4749) Tribe's AtMostOnceInterceptor could
lead to OutOfMemory under heavy load and big messages.
Posted by "Afkham Azeez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AXIS2-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881270#action_12881270 ]
Afkham Azeez commented on AXIS2-4749:
-------------------------------------
Stefano,
Two different messages can have the same hashcode. So, if you simply store the hashcode & compare them, some distinct messages will be detected as duplicates, and will lead to message processing losses.
There is a cleanup task which cleans up this Map, so this Map will not grow infinitely. Also, you could turn off at-most-once message processing if you do not require it using an configuration parameter in the axis2.xml file.
> Tribe's AtMostOnceInterceptor could lead to OutOfMemory under heavy load and big messages.
> ------------------------------------------------------------------------------------------
>
> Key: AXIS2-4749
> URL: https://issues.apache.org/jira/browse/AXIS2-4749
> Project: Axis2
> Issue Type: Bug
> Affects Versions: 1.6
> Environment: Linux 64 bit, JDK 1.6, WSO2 Carbon 2.0.3
> Reporter: Stefano Bruna
>
> Wa are stressing the WSO2 ESB with some cache mediator enabled. Messages are echanged by tribes through the cluster's nodes. Under heavy load and with some big xml messages (1 mb per message) the local variable Map<ChannelMessage, Long> receivedMessages is growing continuously leading to a potential Out of Memory if the cleaning thread that runs every 5 minutes is not fast enough to free up memory.
> See: http://svn.apache.org/viewvc/axis/axis2/java/core/trunk/modules/clustering/src/org/apache/axis2/clustering/tribes/AtMostOnceInterceptor.java?view=markup
> This is because when an object is passed to a HashMap as a key internally the original object is kept. The Hasmap is using a private class Entry that has the hash, the orginal object used to create the hash, and the value that, in this usage case, is a long.
> Somewhere inside HashMap.java
> // private variables
> final Object key;
> Object value;
> Entry next;
> final int hash;
> // constructor
> Entry(int i, Object obj, Object obj1, Entry entry)
> {
> value = obj1; <==== value, ok
> next = entry; <===== entry with the same hash
> key = obj; <======= orginal object, needed for the equals use in case of same hash
> hash = i; <======== key, ok
> }
> So if the AtMostOnceInterceptor manages for example 50 msg/sec and a message is 1 mb we could have within 5 minutes a memory usage for the messageReceived object of 6 GB.
> A simple solution, if we dont'to accept all this as something by desing, could be to pass to the HashMap the already calculated hash of the object (that is also the same method that is called internally int i = hash(obj.hashCode());) to not give the opportunity to the HashMap to keep the actual object used to produce the key.
> .....
> // map
> private static final Map<Integer, Long> receivedMessagesHashCodes =
> new HashMap<Integer, Long>();
> .....
> Integer hashCode = new Integer(msg.hashCode());
>
> if (receivedMessagesHashCodes.get(hashCode) == null) { // If it is a new message, keep track of it
> receivedMessagesHashCodes.put(hashCode, System.currentTimeMillis());
> super.messageReceived(msg);
> } else { // If it is a duplicate message, discard it. i.e. dont call super.messageReceived
> log.info("Duplicate message received from " + TribesUtil.getName(msg.getAddress()));
> }
> etc...
> but maybe is not strong enough in the case tow messages have the same hash. A FIFO queue with a limited capacity ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@axis.apache.org
For additional commands, e-mail: java-dev-help@axis.apache.org