You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@infra.apache.org by "Ted Husted (JIRA)" <ji...@apache.org> on 2007/08/24 14:27:32 UTC
[jira] Created: (INFRA-1334) Intermittent Out Of Memory failures
Intermittent Out Of Memory failures
-----------------------------------
Key: INFRA-1334
URL: https://issues.apache.org/jira/browse/INFRA-1334
Project: Infrastructure
Issue Type: Bug
Security Level: public (Regular issues)
Components: JIRA
Reporter: Ted Husted
[Summarized from an email thread]
Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
Client tools account for half of the current JIRA traffic.
% wc -l issues_weblog
738289
% grep -c soapservice issues_weblog
372529
The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
* http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
is being built straight from the issue tracker.
It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Ian Daniel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528232 ]
Ian Daniel commented on INFRA-1334:
-----------------------------------
I've added instructions to https://support.atlassian.com/browse/JSP-15397.
Kind regards,
Ian
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Jeff Turner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525893 ]
Jeff Turner commented on INFRA-1334:
------------------------------------
The process for diagnosing JIRA OutOfMemoryError causes is very well defined:
http://confluence.atlassian.com/display/JIRA/Causes+of+OutOfMemoryErrors
Just about the only way to diagnose OOMs is to get a heap dump. To that end we should:
- Reduce -Xmx back to 800m (setting it higher pushes the heap dump larger than a 32 bit JVM can load)
- Set -XX:+HeapDumpOnOutOfMemoryError in /usr/local/tomcat/tomcat-jira/bin/setenv.sh
- Wait till the next OOM
- Send the resulting heap dump to Atlassian at https://support.atlassian.com/browse/JSP-15397
If JIRA is OOM'ing that often then turning off SOAP access is a good idea, as it is the most likely culprit.
FYI, we moved from ajax to brutus precisely in order to be able to get heap dumps, which (at the time) could not be obtained on an ia64 architecture. And if you recall, it was Tomcat, not JIRA, that was leaking memory then (http://marc.info/?t=114672636400005&r=1&w=2).
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525983 ]
Henri Yandell commented on INFRA-1334:
--------------------------------------
[Sorry Joe, I meant "short term" in terms of solution, not problem]
I've added those configuration changes to setenv.sh, though not restarted yet. We'll need to turn off the periodic restart Joe to get value from this.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Ted Husted (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Husted updated INFRA-1334:
------------------------------
Security: public (was: non-public)
I've filed with Atlassian as <https://support.atlassian.com/browse/JSP-15397>, and made the ticket public. I know we had a outage on Monday, and I'll try to obtain the logs and add them to the Atlassian ticket.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525510 ]
Henri Yandell commented on INFRA-1334:
--------------------------------------
It seems to have increased in the last week or so. Joe increased the heap memory a few days ago.
Usually the OOM is the JVM short circuiting because too much GC has been happening. Next time it goes down and I notice, I plan to turn off the short circuit feature just to make sure that it's not short circuiting when it shouldn't. Tomcat+JIRA does do a lot of GC anyway.
Is it configured to do heap dumps anywhere at the moment Jeff?
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Joe Schaefer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525515 ]
Joe Schaefer commented on INFRA-1334:
-------------------------------------
We reboot jira daily, so since that GC short-circuiting feature was already
disabled in the live config, and we have since rebooted, it is now turned off.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Jeff Turner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525897 ]
Jeff Turner commented on INFRA-1334:
------------------------------------
Those wishing to participate on the Atlassian support page please register a username at https://support.atlassian.com/secure/Signup!default.jspa and ask Ted to add you to the Cc list.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Yandell closed INFRA-1334.
--------------------------------
Resolution: Fixed
All dealt with - INFRA-1381 records the need to patch the rpc plugin.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
> Assignee: Henri Yandell
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Yandell reassigned INFRA-1334:
------------------------------------
Assignee: Henri Yandell
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
> Assignee: Henri Yandell
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528790 ]
Henri Yandell commented on INFRA-1334:
--------------------------------------
Jed - we'll use this ticket to update the ASF at large about the issue rather than just those who can see the JSP-15397.
Updating, Jed's suggested a change to the rpc plugin to limit memory, so I've made that change and deployed to the main JIRA as of right now.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Joe Schaefer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528042 ]
Joe Schaefer commented on INFRA-1334:
-------------------------------------
Heap dump and catalina.out logs, including a thread dump, are ready.
Unfortunately I can't upload them to the Atlassian ticket because my
account there (joes@apache.org) doesn't have the requisite permissions.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Jed Wesley-Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528683 ]
Jed Wesley-Smith commented on INFRA-1334:
-----------------------------------------
Guys, please see comments on the support issue: https://support.atlassian.com/browse/JSP-15397
Does anyone mind if the discussion of this stays on that ticket? I don't want to spam people with cross-posts or pings on this issue if the other one is changed? Let me know your preference.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528800 ]
Henri Yandell commented on INFRA-1334:
--------------------------------------
I've also turned on user agent logging in the access logs for next time - that way we can better specify what it is that is hitting the rpc badly.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12531843 ]
Henri Yandell commented on INFRA-1334:
--------------------------------------
This is all looking good with a modified RPC plugin that turns off the getProjects method.
I need to integrate that into the upgrade patches and signify that things have been good with Atlassian's ticket.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Ian Daniel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526040 ]
Ian Daniel commented on INFRA-1334:
-----------------------------------
Hi Guys,
We will investigate once you attach a heap dump to https://support.atlassian.com/browse/JSP-15397, as Jeff suggests.
Please be aware that Jeff started four weeks' vacation this week, so we won't have his "insider knowledge" of the setup of your JIRA instance. So please bear that in mind and let us know anything you might think is significant.
Kind regards,
Ian
---
Ian Daniel
Senior Support Engineer, JIRA
Atlassian Software
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525629 ]
Henri Yandell commented on INFRA-1334:
--------------------------------------
#JAVA_OPTS="$JAVA_OPTS -XX:-UseGCOverheadLimit"
Note the '-' sign in the config; the line has to be uncommented to turn off the limit, not commented. So currently it's running.
While rebooting JIRA daily solves the short term issue of JIRA OOMs, it ends any chance of fixing the problem.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1334) Intermittent Out Of Memory failures
Posted by "Joe Schaefer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525785 ]
Joe Schaefer commented on INFRA-1334:
-------------------------------------
I completely disagree with your characterization of the OOM's as a short-term
problem. The reason we even moved jira from ajax to brutus was to deal with
memory leaks, because i386 supposedly had better instrumentation than
ia64. That was over a year ago, nothing has changed except the version number,
and we're now dealing with weekly OOM's which bring down every other service
on brutus.
If restarting jira alleviates the leaks, then trading 30minutes of monthly scheduled
downtime is an major improvement over having a few random OOM outages for
the same timeframe.
> Intermittent Out Of Memory failures
> -----------------------------------
>
> Key: INFRA-1334
> URL: https://issues.apache.org/jira/browse/INFRA-1334
> Project: Infrastructure
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: JIRA
> Reporter: Ted Husted
>
> [Summarized from an email thread]
> Occasionally, JIRA on Brutus will fail. Sometimes, it seems to "spin out of control", knocking off other systems, including Bugzilla. Though, other times, it's been observed that the other JIRA instances are responsive even when one fails.
> It's possible that we are seeing both OOMs and PermGen OOMs. The most recent set were due to some kind of client tool sending a malformed request that ended up requesting huge amounts of data, and consuming large amounts of memory. It's not clear whether the memory usage is a leak or because something that isn't coded to pipeline.
> Client tools account for half of the current JIRA traffic.
> % wc -l issues_weblog
> 738289
> % grep -c soapservice issues_weblog
> 372529
> The underlying problem may be that ill-conceived search requests can consume too much memory and bring the system down. (For other services, like Subversion, we've found ways to keep the CPUs in check.)
> These requests are being made by tools like mylyn, which some developers find very useful. The tools increase integration between development environments and the issue tracker. A page like this
> * http://velocity.apache.org/engine/releases/velocity-1.5/jira-report.html
> is being built straight from the issue tracker.
> It is important that we take steps, since we can't trust a system that's starting to throw OOMs, since it could potentially lead to bad data.
> Since there seems to be more than one type of error condition, whenever a JIRA instance goes down, we should check is whether other instances are responsive. We need, for example, to isolate issues with the httpd front end.
> We might want to look at setting up the Java Service Wrapper (http://wrapper.tanukisoftware.org) for our JVM(s) (its available for both Linux and Solaris). One thing the wrapper can do is monitor the JVM output for such things as OOM messages. Having JWS shut the JVM down and send a mail might help us isolate the problem. If possible, it might be helpful if the wrapper could monitoring JMX events within the monitored JVM, so that the Wrapper can take pro-active action.
> The JIRA searches are based on Lucene. We might want to consider asking one of our ASF Lucene experts to review the Atlassian source code.
> Aside from searches, another cause could be something like DNS resolution via InetAddress? This is a known issue with Java's DNS cache that can cause memory leaks for each IP that hit the server. Any server application that does DNS resolution should be using dnsjava, which has a proper TTL-managed cache. There is also a JVM specific workaround that can be used in the meantime.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.