You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@guacamole.apache.org by James Fraser <Ja...@veritec.com.au> on 2017/08/04 00:15:54 UTC
Server Out Of Memory
Hi All
I recently upgraded to 0.9.13 and am experiencing an issue with my Production server.
This is potentially a Tomcat issue or JDBC driver issue.
If anyone can comment that would be great.
Looking at the Catalina.out file I found
WARNING: The web application [guacamole] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
SEVERE: The web application [guacamole] created a ThreadLocal with key of type [com.google.inject.internal.InjectorImpl$1] (value [com.google.inject.internal.InjectorImpl$1@2c0acdd7]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@2ee3d179]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Aug 02, 2017 8:25:43 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [guacamole] created a ThreadLocal with key of type [com.google.inject.internal.InjectorImpl$1] (value [com.google.inject.internal.InjectorImpl$1@1b5b9da0]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@b5aea9c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Which leads too
Aug 03, 2017 10:04:16 PM org.apache.tomcat.util.net.NioEndpoint$Poller run
SEVERE:
java.lang.OutOfMemoryError: Java heap space
I am just running the standard apt-get configuration of Tomcat 8 on Ubuntu server 16.4
The server has 4GB of ram
Here is the output of Free
root@MGMT-GUAC-01:/var/log/tomcat8# free -h
total used free shared buff/cache available
Mem: 3.4G 939M 128M 22M 2.3G 2.1G
Swap: 0B 0B 0B
A restart of tomcat resolves the issue for a period of time, I have just written a cron job that restarts tomcat on appearance of this issue.
Any advice greatly appreciated.
RE: Server Out Of Memory
Posted by James Fraser <Ja...@veritec.com.au>.
Evening
So I setup a dump for the next time it happened however it failed with permissions on the folder I used.
First error
21:10:19.472 [http-nio-8080-exec-6] ERROR o.a.g.rest.RESTExceptionWrapper - Unexpected internal error:
### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AND user_id = 19
AND permission = 'READ';
SELECT primary_co' at line 15
### The error may exist in org/apache/guacamole/auth/jdbc/connection/ConnectionMapper.xml
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: SELECT guacamole_connection.connection_id, connection_name, parent_id, protocol, max_connections, max_connections_per_user, proxy_hostname, proxy_port, proxy_encryption_method FROM guacamole_connection JOIN guacamole_connection_permission ON guacamole_connection_permission.connection_id = guacamole_connection.connection_id WHERE guacamole_connection.connection_id IN AND user_id = ? AND permission = 'READ'; SELECT primary_connection_id, guacamole_sharing_profile.sharing_profile_id FROM guacamole_sharing_profile JOIN guacamole_sharing_profile_permission ON guacamole_sharing_profile_permission.sharing_profile_id = guacamole_sharing_profile.sharing_profile_id WHERE primary_connection_id IN AND user_id = ? AND permission = 'READ';
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AND user_id = 19
AND permission = 'READ';
Second Error
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /home/NOTREALUSER/java_pid15204.hprof ...
Unable to create /home/NOTREALUSER/java_pid15204.hprof: Permission denied
Aug 07, 2017 9:05:07 PM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor doRun
SEVERE:
java.lang.OutOfMemoryError: Java heap space
at sun.util.resources.TimeZoneNames.getContents(TimeZoneNames.java:285)
at sun.util.resources.OpenListResourceBundle.loadLookup(OpenListResourceBundle.java:137)
at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary(OpenListResourceBundle.java:128)
at sun.util.resources.OpenListResourceBundle.handleKeySet(OpenListResourceBundle.java:96)
at java.util.ResourceBundle.containsKey(ResourceBundle.java:1807)
at sun.util.locale.provider.LocaleResources.getTimeZoneNames(LocaleResources.java:263)
at sun.util.locale.provider.TimeZoneNameProviderImpl.getDisplayNameArray(TimeZoneNameProviderImpl.java:124)
at sun.util.locale.provider.TimeZoneNameProviderImpl.getDisplayName(TimeZoneNameProviderImpl.java:99)
at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getName(TimeZoneNameUtility.java:240)
at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getObject(TimeZoneNameUtility.java:198)
at sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter.getObject(TimeZoneNameUtility.java:184)
at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObjectImpl(LocaleServiceProviderPool.java:281)
at sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObject(LocaleServiceProviderPool.java:265)
at sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayNamesImpl(TimeZoneNameUtility.java:166)
at sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayName(TimeZoneNameUtility.java:137)
at java.util.TimeZone.getDisplayName(TimeZone.java:400)
at java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1271)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:966)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:936)
at java.text.DateFormat.format(DateFormat.java:345)
at org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate(FastHttpDateFormat.java:117)
at org.apache.coyote.http11.AbstractHttp11Processor.prepareResponse(AbstractHttp11Processor.java:1534)
at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:739)
at org.apache.coyote.Response.action(Response.java:179)
at org.apache.coyote.Response.sendHeaders(Response.java:350)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:335)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:317)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:110)
at com.sun.jersey.spi.container.servlet.WebComponent$Writer.flush(WebComponent.java:315)
at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.flush(ContainerResponse.java:145)
at org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1085)
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1606)
Followed by
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
java.lang.OutOfMemoryError: Java heap space
SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
I will adjust the dump for next time.
As noted the machine has 4GB of ram and around 7 users with 3-4 concurrent connections.
In 4 months on 0.9.12 I experienced an issue once which required blipping tomcat but did not delve into it so cannot confirm if the issue was related.
Looking at the glaring error it appears to be JDBC/MYSQL related.
To give some knowledge I did upgrade the database schema and also downloaded the 5.1.43 JDBC driver.
Anyway it is late here in Australia and I need to get some sleep.
Will update tomorrow with MYSQL version etc if requested.
Cheers
JAMES FRASER • MICROSOFT SYSTEMS ENGINEER
-----Original Message-----
From: Mike Jumper [mailto:mike.jumper@guac-dev.org]
Sent: Friday, 4 August 2017 12:05 PM
To: user@guacamole.incubator.apache.org
Subject: Re: Server Out Of Memory
On Thu, Aug 3, 2017 at 6:54 PM, Mike Jumper <mi...@guac-dev.org> wrote:
> On Thu, Aug 3, 2017 at 6:50 PM, James Fraser
> <Ja...@veritec.com.au> wrote:
>> Hi Nick
>>
>> Thanks for your response.
>>
>> After sending off this message I did some digging.
>>
>> I am using JDBC and LDAP auth together.
>>
>> I was digging around the Server Heap error and think that you are on
>> the right track with Xmx value.
>>
>> It was out of the box (from apt-get) set to -Xmx128m, I have adjusted
>> this to 1024m for now and will monitor
>>
>> We concurrently have around 7 users, each user may be accessing 4-5
>> VM’s at once.
>>
>
> Would you be able to take a heap dump to see what is using up so much space?
>
> 7 users is relatively light, and having to manually increase the heap
> shouldn't be necessary in practice. In past versions of Java, they can
> cause more problems than they solve (lengthy GCs), and recent versions
> of Java will ignore these options.
>
Correction: it's permgen that vanished in recent versions of Java, not heap limits.
My other points still stand though. ;)
Re: Server Out Of Memory
Posted by Mike Jumper <mi...@guac-dev.org>.
On Thu, Aug 3, 2017 at 6:54 PM, Mike Jumper <mi...@guac-dev.org> wrote:
> On Thu, Aug 3, 2017 at 6:50 PM, James Fraser
> <Ja...@veritec.com.au> wrote:
>> Hi Nick
>>
>> Thanks for your response.
>>
>> After sending off this message I did some digging.
>>
>> I am using JDBC and LDAP auth together.
>>
>> I was digging around the Server Heap error and think that you are on the
>> right track with Xmx value.
>>
>> It was out of the box (from apt-get) set to -Xmx128m, I have adjusted this
>> to 1024m for now and will monitor
>>
>> We concurrently have around 7 users, each user may be accessing 4-5 VM’s at
>> once.
>>
>
> Would you be able to take a heap dump to see what is using up so much space?
>
> 7 users is relatively light, and having to manually increase the heap
> shouldn't be necessary in practice. In past versions of Java, they can
> cause more problems than they solve (lengthy GCs), and recent versions
> of Java will ignore these options.
>
Correction: it's permgen that vanished in recent versions of Java, not
heap limits.
My other points still stand though. ;)
Re: Server Out Of Memory
Posted by Mike Jumper <mi...@guac-dev.org>.
On Thu, Aug 3, 2017 at 6:50 PM, James Fraser
<Ja...@veritec.com.au> wrote:
> Hi Nick
>
> Thanks for your response.
>
> After sending off this message I did some digging.
>
> I am using JDBC and LDAP auth together.
>
> I was digging around the Server Heap error and think that you are on the
> right track with Xmx value.
>
> It was out of the box (from apt-get) set to -Xmx128m, I have adjusted this
> to 1024m for now and will monitor
>
> We concurrently have around 7 users, each user may be accessing 4-5 VM’s at
> once.
>
Would you be able to take a heap dump to see what is using up so much space?
7 users is relatively light, and having to manually increase the heap
shouldn't be necessary in practice. In past versions of Java, they can
cause more problems than they solve (lengthy GCs), and recent versions
of Java will ignore these options.
- Mike
RE: Server Out Of Memory
Posted by James Fraser <Ja...@veritec.com.au>.
Hi Nick
Thanks for your response.
After sending off this message I did some digging.
I am using JDBC and LDAP auth together.
I was digging around the Server Heap error and think that you are on the right track with Xmx value.
It was out of the box (from apt-get) set to -Xmx128m, I have adjusted this to 1024m for now and will monitor
We concurrently have around 7 users, each user may be accessing 4-5 VM’s at once.
Thanks again for the quick response Nick
James Fraser • Microsoft Systems Engineer
From: Nick Couchman [mailto:nick.couchman@yahoo.com]
Sent: Friday, 4 August 2017 10:52 AM
To: user@guacamole.incubator.apache.org
Subject: Re: Server Out Of Memory
Okay, let me try to take these one at a time...
On Thursday, August 3, 2017, 8:16:09 PM EDT, James Fraser <Ja...@veritec.com.au>> wrote:
> I recently upgraded to 0.9.13 and am experiencing an issue with my Production server.
> This is potentially a Tomcat issue or JDBC driver issue.
What extensions do you have loaded? Looks like MySQL JDBC - anything else?
> WARNING: The web application [guacamole] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
> ...
I use PostgreSQL and see these messages periodically, too, but they've never led to any adverse behavior.
> Which leads too
> Aug 03, 2017 10:04:16 PM org.apache.tomcat.util.net.NioEndpoint$Poller run
> SEVERE:
> java.lang.OutOfMemoryError: Java heap space
Yeah, that's not good, but it doesn't mean your server is running out of memory, it means the JavaVM is running out of heap space. Those are different things. What parameters do you have set for memory in Java in your Tomcat startup? Look for the -Xmx flag either in the ps output for the PID of Java associated with Tomcat or in the Tomcat startup.sh file. If you don't see it, then the default is 1/4 of your total RAM, so 1GB. You can add the -Xmx flag to the java runtime parameters for Tomcat and bump it up to 2GB or something like that and see if that helps. If run out of RAM after bumping it up to 2 or 3GB, then you may have run into a memory leak, but I'd give that a shot, first. When you set it, you can use abbreviations for various byte multiples - for example, -Xmx1024m is 1024MB or 1GB. So, you might want to start with -Xmx2048m to bump up to 2GB and see if that helps.
> The server has 4GB of ram
I ran Guacamole 0.9.12 and the development versions of 0.9.13 on a system with 4GB of RAM for quite some time and never had any issues. How many connections do you have? How many users connecting concurrently?
> root@MGMT-GUAC-01:/var/log/tomcat8# free -h
> total used free shared buff/cache available
> Mem: 3.4G 939M 128M 22M 2.3G 2.1G
> Swap: 0B 0B 0B
> A restart of tomcat resolves the issue for a period of time, I have just written a cron job that restarts tomcat on appearance of this issue.
I've done Linux system admin/engineering for many years, and, from my point of view, those numbers from the output of free look just fine. While it's easy to look at the "free" column and see 128M and think your system is short on RAM, the "available" column is what really counts. Linux uses available RAM to cache and buffer things like disk and network I/O, and your system is consuming 2-ish GB for that. Memory allocated for buffer/cache can be easily freed when applications need it, so that's why the available column shows 2.1GB. So, whenever you ran the "free" command on your system, the system itself is fine on RAM (for the moment) - it's most likely a Java heap size issue (-Xmx flag needs to be set).
-Nick
Re: Server Out Of Memory
Posted by Nick Couchman <ni...@yahoo.com>.
Okay, let me try to take these one at a time...
On Thursday, August 3, 2017, 8:16:09 PM EDT, James Fraser <Ja...@veritec.com.au> wrote:
> I recently upgraded to 0.9.13 and am experiencing an issue with my Production server.
> This is potentially a Tomcat issue or JDBC driver issue.
What extensions do you have loaded? Looks like MySQL JDBC - anything else?
> WARNING: The web application [guacamole] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)> ...
I use PostgreSQL and see these messages periodically, too, but they've never led to any adverse behavior.
> Which leads too
> Aug 03, 2017 10:04:16 PM org.apache.tomcat.util.net.NioEndpoint$Poller run
> SEVERE:
> java.lang.OutOfMemoryError: Java heap space
Yeah, that's not good, but it doesn't mean your server is running out of memory, it means the JavaVM is running out of heap space. Those are different things. What parameters do you have set for memory in Java in your Tomcat startup? Look for the -Xmx flag either in the ps output for the PID of Java associated with Tomcat or in the Tomcat startup.sh file. If you don't see it, then the default is 1/4 of your total RAM, so 1GB. You can add the -Xmx flag to the java runtime parameters for Tomcat and bump it up to 2GB or something like that and see if that helps. If run out of RAM after bumping it up to 2 or 3GB, then you may have run into a memory leak, but I'd give that a shot, first. When you set it, you can use abbreviations for various byte multiples - for example, -Xmx1024m is 1024MB or 1GB. So, you might want to start with -Xmx2048m to bump up to 2GB and see if that helps.
> The server has 4GB of ram
I ran Guacamole 0.9.12 and the development versions of 0.9.13 on a system with 4GB of RAM for quite some time and never had any issues. How many connections do you have? How many users connecting concurrently?
> root@MGMT-GUAC-01:/var/log/tomcat8# free -h
> total used free shared buff/cache available
> Mem: 3.4G 939M 128M 22M 2.3G 2.1G
> Swap: 0B 0B 0B
> A restart of tomcat resolves the issue for a period of time, I have just written a cron job that restarts tomcat on appearance of this issue.
I've done Linux system admin/engineering for many years, and, from my point of view, those numbers from the output of free look just fine. While it's easy to look at the "free" column and see 128M and think your system is short on RAM, the "available" column is what really counts. Linux uses available RAM to cache and buffer things like disk and network I/O, and your system is consuming 2-ish GB for that. Memory allocated for buffer/cache can be easily freed when applications need it, so that's why the available column shows 2.1GB. So, whenever you ran the "free" command on your system, the system itself is fine on RAM (for the moment) - it's most likely a Java heap size issue (-Xmx flag needs to be set).
-Nick