You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Jeff Huang (JIRA)" <ji...@apache.org> on 2012/11/20 16:43:01 UTC
[jira] [Created] (AMQ-4182) Memory Leak for ActiveMQBytesMessage
with Compression as true
Jeff Huang created AMQ-4182:
-------------------------------
Summary: Memory Leak for ActiveMQBytesMessage with Compression as true
Key: AMQ-4182
URL: https://issues.apache.org/jira/browse/AMQ-4182
Project: ActiveMQ
Issue Type: Bug
Components: JMS client
Affects Versions: 5.5.1
Environment: Linux(Redhat 5.5), Windows 7
Reporter: Jeff Huang
InflaterInputStream is supposed to close explicitly to release resource allocated by its JNI methods. In ActiveMQBytesMessage, dataIn property is disposed simply without closing it, which results in some weird memory leak that can't be detected from heap size. It can't be controlled by -Xmx or -XX:MaxDirectMemorySize.
Please run the following test program to verify the issue:
import java.util.concurrent.TimeUnit;
import javax.jms.BytesMessage;
import javax.jms.Connection;
import javax.jms.Session;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.command.ActiveMQBytesMessage;
/**
* A simple test to verify memory leak in ActiveMQBytesMessage.
*/
public class Main
{
public static void main(String[] args) throws Exception
{
ActiveMQConnectionFactory connFactory = new ActiveMQConnectionFactory("vm://localhost");
connFactory.setUseCompression(true);
Connection conn = connFactory.createConnection();
Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
BytesMessage message = session.createBytesMessage();
message.writeBytes(new byte[1024]);
ActiveMQBytesMessage m = (ActiveMQBytesMessage)message;
if(!m.isCompressed())
{
throw new RuntimeException();
}
while (true)
{
for (int k = 0; k < 1024; ++k)
{
message.reset();
byte[] data = new byte[1024];
message.readBytes(data);
}
TimeUnit.MILLISECONDS.sleep(10);
}
}
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-4182) Memory Leak for ActiveMQBytesMessage
with Compression as true
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506793#comment-13506793 ]
Timothy Bish commented on AMQ-4182:
-----------------------------------
You're welcome to submit a patch if you can see a reasonable solution to the issue you are seeing.
> Memory Leak for ActiveMQBytesMessage with Compression as true
> -------------------------------------------------------------
>
> Key: AMQ-4182
> URL: https://issues.apache.org/jira/browse/AMQ-4182
> Project: ActiveMQ
> Issue Type: Bug
> Components: JMS client
> Affects Versions: 5.5.1
> Environment: Linux(Redhat 5.5), Windows 7
> Reporter: Jeff Huang
> Priority: Minor
>
> InflaterInputStream is supposed to close explicitly to release resource allocated by its JNI methods. In ActiveMQBytesMessage, dataIn property is disposed simply without closing it, which results in some weird memory leak that can't be detected from heap size. It can't be controlled by -Xmx or -XX:MaxDirectMemorySize.
> Please run the following test program to verify the issue:
> import java.util.concurrent.TimeUnit;
> import javax.jms.BytesMessage;
> import javax.jms.Connection;
> import javax.jms.Session;
> import org.apache.activemq.ActiveMQConnectionFactory;
> import org.apache.activemq.command.ActiveMQBytesMessage;
> /**
> * A simple test to verify memory leak in ActiveMQBytesMessage.
> */
> public class Main
> {
> public static void main(String[] args) throws Exception
> {
> ActiveMQConnectionFactory connFactory = new ActiveMQConnectionFactory("vm://localhost");
> connFactory.setUseCompression(true);
> Connection conn = connFactory.createConnection();
> Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
> BytesMessage message = session.createBytesMessage();
>
> message.writeBytes(new byte[1024]);
> ActiveMQBytesMessage m = (ActiveMQBytesMessage)message;
> if(!m.isCompressed())
> {
> throw new RuntimeException();
> }
>
>
> while (true)
> {
> for (int k = 0; k < 1024; ++k)
> {
> message.reset();
> byte[] data = new byte[1024];
> message.readBytes(data);
> }
> TimeUnit.MILLISECONDS.sleep(10);
> }
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-4182) Memory Leak for ActiveMQBytesMessage
with Compression as true
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish updated AMQ-4182:
------------------------------
Priority: Minor (was: Major)
> Memory Leak for ActiveMQBytesMessage with Compression as true
> -------------------------------------------------------------
>
> Key: AMQ-4182
> URL: https://issues.apache.org/jira/browse/AMQ-4182
> Project: ActiveMQ
> Issue Type: Bug
> Components: JMS client
> Affects Versions: 5.5.1
> Environment: Linux(Redhat 5.5), Windows 7
> Reporter: Jeff Huang
> Priority: Minor
>
> InflaterInputStream is supposed to close explicitly to release resource allocated by its JNI methods. In ActiveMQBytesMessage, dataIn property is disposed simply without closing it, which results in some weird memory leak that can't be detected from heap size. It can't be controlled by -Xmx or -XX:MaxDirectMemorySize.
> Please run the following test program to verify the issue:
> import java.util.concurrent.TimeUnit;
> import javax.jms.BytesMessage;
> import javax.jms.Connection;
> import javax.jms.Session;
> import org.apache.activemq.ActiveMQConnectionFactory;
> import org.apache.activemq.command.ActiveMQBytesMessage;
> /**
> * A simple test to verify memory leak in ActiveMQBytesMessage.
> */
> public class Main
> {
> public static void main(String[] args) throws Exception
> {
> ActiveMQConnectionFactory connFactory = new ActiveMQConnectionFactory("vm://localhost");
> connFactory.setUseCompression(true);
> Connection conn = connFactory.createConnection();
> Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
> BytesMessage message = session.createBytesMessage();
>
> message.writeBytes(new byte[1024]);
> ActiveMQBytesMessage m = (ActiveMQBytesMessage)message;
> if(!m.isCompressed())
> {
> throw new RuntimeException();
> }
>
>
> while (true)
> {
> for (int k = 0; k < 1024; ++k)
> {
> message.reset();
> byte[] data = new byte[1024];
> message.readBytes(data);
> }
> TimeUnit.MILLISECONDS.sleep(10);
> }
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-4182) Memory Leak for ActiveMQBytesMessage
with Compression as true
Posted by "Jeff Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506880#comment-13506880 ]
Jeff Huang commented on AMQ-4182:
---------------------------------
Unfortunately, I don't have a solution for that. I would tend to say that there is not a valid solution for that.
We could try to add a close() or release() method to ActiveMQBytesMessage, but I don't think the users are going to call it because it is not part of the JMS API.
Or we can try to find another pure Java compression library to avoid the need to release the resource explicitly. But that may probably mean we need to sacrifice some performance when doing compression.
Or we may want to decompress the whole buffer, instead of remaining a stream for it. The memory footprint would be bigger, but comparing to blowing out the whole system, this is acceptable.
Or we can call off this feature. Tell the users not to use BytesMessage in ActiveMQ when they try to use compression.
> Memory Leak for ActiveMQBytesMessage with Compression as true
> -------------------------------------------------------------
>
> Key: AMQ-4182
> URL: https://issues.apache.org/jira/browse/AMQ-4182
> Project: ActiveMQ
> Issue Type: Bug
> Components: JMS client
> Affects Versions: 5.5.1
> Environment: Linux(Redhat 5.5), Windows 7
> Reporter: Jeff Huang
> Priority: Minor
>
> InflaterInputStream is supposed to close explicitly to release resource allocated by its JNI methods. In ActiveMQBytesMessage, dataIn property is disposed simply without closing it, which results in some weird memory leak that can't be detected from heap size. It can't be controlled by -Xmx or -XX:MaxDirectMemorySize.
> Please run the following test program to verify the issue:
> import java.util.concurrent.TimeUnit;
> import javax.jms.BytesMessage;
> import javax.jms.Connection;
> import javax.jms.Session;
> import org.apache.activemq.ActiveMQConnectionFactory;
> import org.apache.activemq.command.ActiveMQBytesMessage;
> /**
> * A simple test to verify memory leak in ActiveMQBytesMessage.
> */
> public class Main
> {
> public static void main(String[] args) throws Exception
> {
> ActiveMQConnectionFactory connFactory = new ActiveMQConnectionFactory("vm://localhost");
> connFactory.setUseCompression(true);
> Connection conn = connFactory.createConnection();
> Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
> BytesMessage message = session.createBytesMessage();
>
> message.writeBytes(new byte[1024]);
> ActiveMQBytesMessage m = (ActiveMQBytesMessage)message;
> if(!m.isCompressed())
> {
> throw new RuntimeException();
> }
>
>
> while (true)
> {
> for (int k = 0; k < 1024; ++k)
> {
> message.reset();
> byte[] data = new byte[1024];
> message.readBytes(data);
> }
> TimeUnit.MILLISECONDS.sleep(10);
> }
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-4182) Memory Leak for ActiveMQBytesMessage
with Compression as true
Posted by "Jeff Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506175#comment-13506175 ]
Jeff Huang commented on AMQ-4182:
---------------------------------
I read the changes. Fix in reset() is good I believe. But I am worried about the finalize() method because I don't think it would work as good as we would expect. Please take a look at how Sun's JDK 1.6 implemented java.util.zip.Inflater
336 /**
337 * Closes the decompressor and discards any unprocessed input.
338 * This method should be called when the decompressor is no longer
339 * being used, but will also be called automatically by the finalize()
340 * method. Once this method is called, the behavior of the Inflater
341 * object is undefined.
342 */
343 public void end() {
344 synchronized (zsRef) {
345 long addr = zsRef.address();
346 zsRef.clear();
347 if (addr != 0) {
348 end(addr);
349 buf = null;
350 }
351 }
352 }
353
354 /**
355 * Closes the decompressor when garbage is collected.
356 */
357 protected void finalize() {
358 end();
359 }
They have already done finalize() with end(). So if finalize() works well, the problem wouldn't exist in the first place. And since those memory is allocated out of JVM heap, the GC could have problem to sense the necessity to clean something even the system blows out.
> Memory Leak for ActiveMQBytesMessage with Compression as true
> -------------------------------------------------------------
>
> Key: AMQ-4182
> URL: https://issues.apache.org/jira/browse/AMQ-4182
> Project: ActiveMQ
> Issue Type: Bug
> Components: JMS client
> Affects Versions: 5.5.1
> Environment: Linux(Redhat 5.5), Windows 7
> Reporter: Jeff Huang
>
> InflaterInputStream is supposed to close explicitly to release resource allocated by its JNI methods. In ActiveMQBytesMessage, dataIn property is disposed simply without closing it, which results in some weird memory leak that can't be detected from heap size. It can't be controlled by -Xmx or -XX:MaxDirectMemorySize.
> Please run the following test program to verify the issue:
> import java.util.concurrent.TimeUnit;
> import javax.jms.BytesMessage;
> import javax.jms.Connection;
> import javax.jms.Session;
> import org.apache.activemq.ActiveMQConnectionFactory;
> import org.apache.activemq.command.ActiveMQBytesMessage;
> /**
> * A simple test to verify memory leak in ActiveMQBytesMessage.
> */
> public class Main
> {
> public static void main(String[] args) throws Exception
> {
> ActiveMQConnectionFactory connFactory = new ActiveMQConnectionFactory("vm://localhost");
> connFactory.setUseCompression(true);
> Connection conn = connFactory.createConnection();
> Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
> BytesMessage message = session.createBytesMessage();
>
> message.writeBytes(new byte[1024]);
> ActiveMQBytesMessage m = (ActiveMQBytesMessage)message;
> if(!m.isCompressed())
> {
> throw new RuntimeException();
> }
>
>
> while (true)
> {
> for (int k = 0; k < 1024; ++k)
> {
> message.reset();
> byte[] data = new byte[1024];
> message.readBytes(data);
> }
> TimeUnit.MILLISECONDS.sleep(10);
> }
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (AMQ-4182) Memory Leak for ActiveMQBytesMessage
with Compression as true
Posted by "Jeff Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Huang reopened AMQ-4182:
-----------------------------
The change on finalize() won't fix the problem because java.util.zip.Inflater tries that already. The memory problem happens out of the normal heap and GC won't detect the memory shortage, in turn objects won't be wiped out at all.
> Memory Leak for ActiveMQBytesMessage with Compression as true
> -------------------------------------------------------------
>
> Key: AMQ-4182
> URL: https://issues.apache.org/jira/browse/AMQ-4182
> Project: ActiveMQ
> Issue Type: Bug
> Components: JMS client
> Affects Versions: 5.5.1
> Environment: Linux(Redhat 5.5), Windows 7
> Reporter: Jeff Huang
>
> InflaterInputStream is supposed to close explicitly to release resource allocated by its JNI methods. In ActiveMQBytesMessage, dataIn property is disposed simply without closing it, which results in some weird memory leak that can't be detected from heap size. It can't be controlled by -Xmx or -XX:MaxDirectMemorySize.
> Please run the following test program to verify the issue:
> import java.util.concurrent.TimeUnit;
> import javax.jms.BytesMessage;
> import javax.jms.Connection;
> import javax.jms.Session;
> import org.apache.activemq.ActiveMQConnectionFactory;
> import org.apache.activemq.command.ActiveMQBytesMessage;
> /**
> * A simple test to verify memory leak in ActiveMQBytesMessage.
> */
> public class Main
> {
> public static void main(String[] args) throws Exception
> {
> ActiveMQConnectionFactory connFactory = new ActiveMQConnectionFactory("vm://localhost");
> connFactory.setUseCompression(true);
> Connection conn = connFactory.createConnection();
> Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
> BytesMessage message = session.createBytesMessage();
>
> message.writeBytes(new byte[1024]);
> ActiveMQBytesMessage m = (ActiveMQBytesMessage)message;
> if(!m.isCompressed())
> {
> throw new RuntimeException();
> }
>
>
> while (true)
> {
> for (int k = 0; k < 1024; ++k)
> {
> message.reset();
> byte[] data = new byte[1024];
> message.readBytes(data);
> }
> TimeUnit.MILLISECONDS.sleep(10);
> }
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-4182) Memory Leak for ActiveMQBytesMessage
with Compression as true
Posted by "Jeff Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506194#comment-13506194 ]
Jeff Huang commented on AMQ-4182:
---------------------------------
Please try to run the test program with -Xmx10M, -X128M, and -X2G, and monitor the JVM memory usage. Here are the results in my own test system with 8G memory when it was stable:
-Xmx10M
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15070 root 17 0 429m 123m 9856 S 80.2 1.5 0:48.49 java
-Xmx128M
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15115 root 17 0 2808m 2.0g 9856 S 78.6 26.0 0:22.70 java
-X2G
Never stable and the whole system just ran out of memory.
That means when -Xmx is big enough for holding NORMAL objects the GC just won't see the memory problem at all, which results in the finalize() method won't be executed ever.
> Memory Leak for ActiveMQBytesMessage with Compression as true
> -------------------------------------------------------------
>
> Key: AMQ-4182
> URL: https://issues.apache.org/jira/browse/AMQ-4182
> Project: ActiveMQ
> Issue Type: Bug
> Components: JMS client
> Affects Versions: 5.5.1
> Environment: Linux(Redhat 5.5), Windows 7
> Reporter: Jeff Huang
>
> InflaterInputStream is supposed to close explicitly to release resource allocated by its JNI methods. In ActiveMQBytesMessage, dataIn property is disposed simply without closing it, which results in some weird memory leak that can't be detected from heap size. It can't be controlled by -Xmx or -XX:MaxDirectMemorySize.
> Please run the following test program to verify the issue:
> import java.util.concurrent.TimeUnit;
> import javax.jms.BytesMessage;
> import javax.jms.Connection;
> import javax.jms.Session;
> import org.apache.activemq.ActiveMQConnectionFactory;
> import org.apache.activemq.command.ActiveMQBytesMessage;
> /**
> * A simple test to verify memory leak in ActiveMQBytesMessage.
> */
> public class Main
> {
> public static void main(String[] args) throws Exception
> {
> ActiveMQConnectionFactory connFactory = new ActiveMQConnectionFactory("vm://localhost");
> connFactory.setUseCompression(true);
> Connection conn = connFactory.createConnection();
> Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
> BytesMessage message = session.createBytesMessage();
>
> message.writeBytes(new byte[1024]);
> ActiveMQBytesMessage m = (ActiveMQBytesMessage)message;
> if(!m.isCompressed())
> {
> throw new RuntimeException();
> }
>
>
> while (true)
> {
> for (int k = 0; k < 1024; ++k)
> {
> message.reset();
> byte[] data = new byte[1024];
> message.readBytes(data);
> }
> TimeUnit.MILLISECONDS.sleep(10);
> }
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AMQ-4182) Memory Leak for ActiveMQBytesMessage
with Compression as true
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish resolved AMQ-4182.
-------------------------------
Resolution: Fixed
Added some additional logic to attempt to close the dataIn stream when possible.
> Memory Leak for ActiveMQBytesMessage with Compression as true
> -------------------------------------------------------------
>
> Key: AMQ-4182
> URL: https://issues.apache.org/jira/browse/AMQ-4182
> Project: ActiveMQ
> Issue Type: Bug
> Components: JMS client
> Affects Versions: 5.5.1
> Environment: Linux(Redhat 5.5), Windows 7
> Reporter: Jeff Huang
>
> InflaterInputStream is supposed to close explicitly to release resource allocated by its JNI methods. In ActiveMQBytesMessage, dataIn property is disposed simply without closing it, which results in some weird memory leak that can't be detected from heap size. It can't be controlled by -Xmx or -XX:MaxDirectMemorySize.
> Please run the following test program to verify the issue:
> import java.util.concurrent.TimeUnit;
> import javax.jms.BytesMessage;
> import javax.jms.Connection;
> import javax.jms.Session;
> import org.apache.activemq.ActiveMQConnectionFactory;
> import org.apache.activemq.command.ActiveMQBytesMessage;
> /**
> * A simple test to verify memory leak in ActiveMQBytesMessage.
> */
> public class Main
> {
> public static void main(String[] args) throws Exception
> {
> ActiveMQConnectionFactory connFactory = new ActiveMQConnectionFactory("vm://localhost");
> connFactory.setUseCompression(true);
> Connection conn = connFactory.createConnection();
> Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
> BytesMessage message = session.createBytesMessage();
>
> message.writeBytes(new byte[1024]);
> ActiveMQBytesMessage m = (ActiveMQBytesMessage)message;
> if(!m.isCompressed())
> {
> throw new RuntimeException();
> }
>
>
> while (true)
> {
> for (int k = 0; k < 1024; ++k)
> {
> message.reset();
> byte[] data = new byte[1024];
> message.readBytes(data);
> }
> TimeUnit.MILLISECONDS.sleep(10);
> }
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira