You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@esme.apache.org by Richard Hirsch <hi...@gmail.com> on 2009/06/22 08:59:38 UTC
Memory usage -> stability probs?
Hi,
After the latest deployment on stax, we've been having a few stability
problems where CPU and memory max out (see attached picture) with very few
users. A restart is necessary to be able to use the app again.
Maybe, someone could check to see if we have some sort of stability issues.
Thanks.
D.
Re: Memory usage -> stability probs?
Posted by Ethan Jewett <es...@gmail.com>.
I did set up my usual "every 5 minutes" rss checking action yesterday
afternoon (US time). Not sure if that might have made a difference.
Ethan
On Mon, Jun 22, 2009 at 1:53 PM, David Pollak <feeder.of.the.bears@gmail.com
> wrote:
> I'll run an instance in a VM Ware instance with restraint memory and see if
> I can make things go bad
>
> On Mon, Jun 22, 2009 at 10:41 AM, Richard Hirsch <hirsch.dick@gmail.com
> >wrote:
>
> > Daniel thought it might be associated with actions but wasn't sure.
> >
> > There were less than 4 users in the db and just two users working
> > concurrently. I checked the trace files and didn't see anything.
> >
> > I'll have to watch it more clsely to see if I see any patterns.
> >
> > D.
> > On Mon, Jun 22, 2009 at 5:29 PM, David Pollak <
> > feeder.of.the.bears@gmail.com
> > > wrote:
> >
> > > What kind of user interaction is required to cause the problem or "just
> > > running it"?
> > >
> > > On Sun, Jun 21, 2009 at 11:59 PM, Richard Hirsch <
> hirsch.dick@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > After the latest deployment on stax, we've been having a few
> stability
> > > > problems where CPU and memory max out (see attached picture) with
> very
> > > few
> > > > users. A restart is necessary to be able to use the app again.
> > > >
> > > > Maybe, someone could check to see if we have some sort of stability
> > > issues.
> > > >
> > > > Thanks.
> > > >
> > > > D.
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Lift, the simply functional web framework http://liftweb.net
> > > Beginning Scala http://www.apress.com/book/view/1430219890
> > > Follow me: http://twitter.com/dpp
> > > Git some: http://github.com/dpp
> > >
> >
>
>
>
> --
> Lift, the simply functional web framework http://liftweb.net
> Beginning Scala http://www.apress.com/book/view/1430219890
> Follow me: http://twitter.com/dpp
> Git some: http://github.com/dpp
>
Re: Memory usage -> stability probs?
Posted by David Pollak <fe...@gmail.com>.
I'll run an instance in a VM Ware instance with restraint memory and see if
I can make things go bad
On Mon, Jun 22, 2009 at 10:41 AM, Richard Hirsch <hi...@gmail.com>wrote:
> Daniel thought it might be associated with actions but wasn't sure.
>
> There were less than 4 users in the db and just two users working
> concurrently. I checked the trace files and didn't see anything.
>
> I'll have to watch it more clsely to see if I see any patterns.
>
> D.
> On Mon, Jun 22, 2009 at 5:29 PM, David Pollak <
> feeder.of.the.bears@gmail.com
> > wrote:
>
> > What kind of user interaction is required to cause the problem or "just
> > running it"?
> >
> > On Sun, Jun 21, 2009 at 11:59 PM, Richard Hirsch <hirsch.dick@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > After the latest deployment on stax, we've been having a few stability
> > > problems where CPU and memory max out (see attached picture) with very
> > few
> > > users. A restart is necessary to be able to use the app again.
> > >
> > > Maybe, someone could check to see if we have some sort of stability
> > issues.
> > >
> > > Thanks.
> > >
> > > D.
> > >
> > >
> >
> >
> >
> > --
> > Lift, the simply functional web framework http://liftweb.net
> > Beginning Scala http://www.apress.com/book/view/1430219890
> > Follow me: http://twitter.com/dpp
> > Git some: http://github.com/dpp
> >
>
--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp
Re: Memory usage -> stability probs?
Posted by David Pollak <fe...@gmail.com>.
On Mon, Jun 22, 2009 at 2:03 PM, David Pollak <feeder.of.the.bears@gmail.com
> wrote:
> I added code that will dump memory information every 10 minutes. Please
> put the new code live. If it barfs, please send me the output (privately is
> better so we don't spam the list with a 10MB file)
> I'm running an instance on my local box and will see if I can find memory
> leaks.
>
I haven't seen any memory leaks.
>
>
> On Mon, Jun 22, 2009 at 10:41 AM, Richard Hirsch <hi...@gmail.com>wrote:
>
>> Daniel thought it might be associated with actions but wasn't sure.
>>
>> There were less than 4 users in the db and just two users working
>> concurrently. I checked the trace files and didn't see anything.
>>
>> I'll have to watch it more clsely to see if I see any patterns.
>>
>> D.
>> On Mon, Jun 22, 2009 at 5:29 PM, David Pollak <
>> feeder.of.the.bears@gmail.com
>> > wrote:
>>
>> > What kind of user interaction is required to cause the problem or "just
>> > running it"?
>> >
>> > On Sun, Jun 21, 2009 at 11:59 PM, Richard Hirsch <hirsch.dick@gmail.com
>> > >wrote:
>> >
>> > > Hi,
>> > >
>> > > After the latest deployment on stax, we've been having a few stability
>> > > problems where CPU and memory max out (see attached picture) with very
>> > few
>> > > users. A restart is necessary to be able to use the app again.
>> > >
>> > > Maybe, someone could check to see if we have some sort of stability
>> > issues.
>> > >
>> > > Thanks.
>> > >
>> > > D.
>> > >
>> > >
>> >
>> >
>> >
>> > --
>> > Lift, the simply functional web framework http://liftweb.net
>> > Beginning Scala http://www.apress.com/book/view/1430219890
>> > Follow me: http://twitter.com/dpp
>> > Git some: http://github.com/dpp
>> >
>>
>
>
>
> --
> Lift, the simply functional web framework http://liftweb.net
> Beginning Scala http://www.apress.com/book/view/1430219890
> Follow me: http://twitter.com/dpp
> Git some: http://github.com/dpp
>
--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp
Re: Memory usage -> stability probs?
Posted by Richard Hirsch <hi...@gmail.com>.
I don't know whether we'll have access to the thread dumps, etc in the stax
environment. Usually, we just have access to performance-related info via
the stax operations diagrams / UI.
I'll ask the people at stax if there is someway to get this data.
D.
On Tue, Jun 23, 2009 at 1:36 PM, Erik Engbrecht <er...@gmail.com>wrote:
> -XX:+*HeapDumpOnOutOfMemoryError** **is also very useful for post-mortems.*
>
> On Tue, Jun 23, 2009 at 2:46 AM, Vassil Dichev <vd...@apache.org> wrote:
>
> > > I added code that will dump memory information every 10 minutes.
> Please
> > put
> > > the new code live. If it barfs, please send me the output (privately
> is
> > > better so we don't spam the list with a 10MB file)
> > > I'm running an instance on my local box and will see if I can find
> memory
> > > leaks.
> >
> > This is cool, but I'm afraid we might need some extra info from the
> > JVM. I'm not sure you can change the command-line or environment you
> > use to start the application in Stax, but this should display detailed
> > information for the Sun JVM.
> >
> > MAVEN_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintClassHistogram
> >
> > and for extra debugging, you can turn on the built in profiler with
> > the following options:
> >
> > -Xrunhprof:heap=sites,cpu=samples,doe=y
> >
> > It will make the JVM run even slower, but could really pinpoint where
> > the problem was.
> >
> > As for thread dumps, they're even more useful if you encounter the
> > 100% CPU issue, but we need to send a signal to the JVM, which I'm not
> > sure you can do at all in a managed stax environment. There are
> > solutions to trigger it programmatically, but you need to include a
> > native library, which is another restriction we might not be able to
> > overcome.
> >
> > All in all, it would be much better if we can reproduce it locally,
> > where we can play with the application. I assume you couldn't set
> > MAVEN_OPTS last time there was a problem, since there was no feedback.
> > Monitoring an app for memory problems in a restricted environment is
> > notoriously hard.
> >
> > Vassil
> >
>
>
>
> --
> http://erikengbrecht.blogspot.com/
>
Re: Memory usage -> stability probs?
Posted by Daniel Koller <da...@googlemail.com>.
...to be more specific regarding Dick comment on my experience:
- the server (on stax + my local machine) got extreme performance issues
when I wanted to enter an actor definition in the UI.
- Sending the form seemed not to work (at least there was no visual
feedback, whether it worked), and
- the new action never showed up in the list.
- After a restart of the app, the action admin interface was available
again.
(I found this out, when I prototyped AMQP connectivity, which involved the
insertion of a new action class.)
Kind regards,
Daniel
--
---
Daniel Koller
Jahnstrasse 20
80469 München * dakoller@googlemail.com
Re: Memory usage -> stability probs?
Posted by Erik Engbrecht <er...@gmail.com>.
Here's a really useful reference:
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp*
*See the "debugging options" section toward the bottom.
On Tue, Jun 23, 2009 at 9:15 AM, David Pollak <feeder.of.the.bears@gmail.com
> wrote:
> Vassil & Erik -- lots of useful stuff in here... I wish I had know about
> these options when I was debugging the Actors memory retention problems...
> :-)
>
> On Tue, Jun 23, 2009 at 4:36 AM, Erik Engbrecht <erik.engbrecht@gmail.com
> >wrote:
>
> > -XX:+*HeapDumpOnOutOfMemoryError** **is also very useful for
> post-mortems.*
> >
> > On Tue, Jun 23, 2009 at 2:46 AM, Vassil Dichev <vd...@apache.org>
> wrote:
> >
> > > > I added code that will dump memory information every 10 minutes.
> > Please
> > > put
> > > > the new code live. If it barfs, please send me the output (privately
> > is
> > > > better so we don't spam the list with a 10MB file)
> > > > I'm running an instance on my local box and will see if I can find
> > memory
> > > > leaks.
> > >
> > > This is cool, but I'm afraid we might need some extra info from the
> > > JVM. I'm not sure you can change the command-line or environment you
> > > use to start the application in Stax, but this should display detailed
> > > information for the Sun JVM.
> > >
> > > MAVEN_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintClassHistogram
> > >
> > > and for extra debugging, you can turn on the built in profiler with
> > > the following options:
> > >
> > > -Xrunhprof:heap=sites,cpu=samples,doe=y
> > >
> > > It will make the JVM run even slower, but could really pinpoint where
> > > the problem was.
> > >
> > > As for thread dumps, they're even more useful if you encounter the
> > > 100% CPU issue, but we need to send a signal to the JVM, which I'm not
> > > sure you can do at all in a managed stax environment. There are
> > > solutions to trigger it programmatically, but you need to include a
> > > native library, which is another restriction we might not be able to
> > > overcome.
> > >
> > > All in all, it would be much better if we can reproduce it locally,
> > > where we can play with the application. I assume you couldn't set
> > > MAVEN_OPTS last time there was a problem, since there was no feedback.
> > > Monitoring an app for memory problems in a restricted environment is
> > > notoriously hard.
> > >
> > > Vassil
> > >
> >
> >
> >
> > --
> > http://erikengbrecht.blogspot.com/
> >
>
>
>
> --
> Lift, the simply functional web framework http://liftweb.net
> Beginning Scala http://www.apress.com/book/view/1430219890
> Follow me: http://twitter.com/dpp
> Git some: http://github.com/dpp
>
--
http://erikengbrecht.blogspot.com/
Re: Memory usage -> stability probs?
Posted by David Pollak <fe...@gmail.com>.
Vassil & Erik -- lots of useful stuff in here... I wish I had know about
these options when I was debugging the Actors memory retention problems...
:-)
On Tue, Jun 23, 2009 at 4:36 AM, Erik Engbrecht <er...@gmail.com>wrote:
> -XX:+*HeapDumpOnOutOfMemoryError** **is also very useful for post-mortems.*
>
> On Tue, Jun 23, 2009 at 2:46 AM, Vassil Dichev <vd...@apache.org> wrote:
>
> > > I added code that will dump memory information every 10 minutes.
> Please
> > put
> > > the new code live. If it barfs, please send me the output (privately
> is
> > > better so we don't spam the list with a 10MB file)
> > > I'm running an instance on my local box and will see if I can find
> memory
> > > leaks.
> >
> > This is cool, but I'm afraid we might need some extra info from the
> > JVM. I'm not sure you can change the command-line or environment you
> > use to start the application in Stax, but this should display detailed
> > information for the Sun JVM.
> >
> > MAVEN_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintClassHistogram
> >
> > and for extra debugging, you can turn on the built in profiler with
> > the following options:
> >
> > -Xrunhprof:heap=sites,cpu=samples,doe=y
> >
> > It will make the JVM run even slower, but could really pinpoint where
> > the problem was.
> >
> > As for thread dumps, they're even more useful if you encounter the
> > 100% CPU issue, but we need to send a signal to the JVM, which I'm not
> > sure you can do at all in a managed stax environment. There are
> > solutions to trigger it programmatically, but you need to include a
> > native library, which is another restriction we might not be able to
> > overcome.
> >
> > All in all, it would be much better if we can reproduce it locally,
> > where we can play with the application. I assume you couldn't set
> > MAVEN_OPTS last time there was a problem, since there was no feedback.
> > Monitoring an app for memory problems in a restricted environment is
> > notoriously hard.
> >
> > Vassil
> >
>
>
>
> --
> http://erikengbrecht.blogspot.com/
>
--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp
Re: Memory usage -> stability probs?
Posted by Erik Engbrecht <er...@gmail.com>.
-XX:+*HeapDumpOnOutOfMemoryError** **is also very useful for post-mortems.*
On Tue, Jun 23, 2009 at 2:46 AM, Vassil Dichev <vd...@apache.org> wrote:
> > I added code that will dump memory information every 10 minutes. Please
> put
> > the new code live. If it barfs, please send me the output (privately is
> > better so we don't spam the list with a 10MB file)
> > I'm running an instance on my local box and will see if I can find memory
> > leaks.
>
> This is cool, but I'm afraid we might need some extra info from the
> JVM. I'm not sure you can change the command-line or environment you
> use to start the application in Stax, but this should display detailed
> information for the Sun JVM.
>
> MAVEN_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintClassHistogram
>
> and for extra debugging, you can turn on the built in profiler with
> the following options:
>
> -Xrunhprof:heap=sites,cpu=samples,doe=y
>
> It will make the JVM run even slower, but could really pinpoint where
> the problem was.
>
> As for thread dumps, they're even more useful if you encounter the
> 100% CPU issue, but we need to send a signal to the JVM, which I'm not
> sure you can do at all in a managed stax environment. There are
> solutions to trigger it programmatically, but you need to include a
> native library, which is another restriction we might not be able to
> overcome.
>
> All in all, it would be much better if we can reproduce it locally,
> where we can play with the application. I assume you couldn't set
> MAVEN_OPTS last time there was a problem, since there was no feedback.
> Monitoring an app for memory problems in a restricted environment is
> notoriously hard.
>
> Vassil
>
--
http://erikengbrecht.blogspot.com/
Re: Memory usage -> stability probs?
Posted by Vassil Dichev <vd...@apache.org>.
> I added code that will dump memory information every 10 minutes. Please put
> the new code live. If it barfs, please send me the output (privately is
> better so we don't spam the list with a 10MB file)
> I'm running an instance on my local box and will see if I can find memory
> leaks.
This is cool, but I'm afraid we might need some extra info from the
JVM. I'm not sure you can change the command-line or environment you
use to start the application in Stax, but this should display detailed
information for the Sun JVM.
MAVEN_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintClassHistogram
and for extra debugging, you can turn on the built in profiler with
the following options:
-Xrunhprof:heap=sites,cpu=samples,doe=y
It will make the JVM run even slower, but could really pinpoint where
the problem was.
As for thread dumps, they're even more useful if you encounter the
100% CPU issue, but we need to send a signal to the JVM, which I'm not
sure you can do at all in a managed stax environment. There are
solutions to trigger it programmatically, but you need to include a
native library, which is another restriction we might not be able to
overcome.
All in all, it would be much better if we can reproduce it locally,
where we can play with the application. I assume you couldn't set
MAVEN_OPTS last time there was a problem, since there was no feedback.
Monitoring an app for memory problems in a restricted environment is
notoriously hard.
Vassil
Re: Memory usage -> stability probs?
Posted by David Pollak <fe...@gmail.com>.
I added code that will dump memory information every 10 minutes. Please put
the new code live. If it barfs, please send me the output (privately is
better so we don't spam the list with a 10MB file)
I'm running an instance on my local box and will see if I can find memory
leaks.
On Mon, Jun 22, 2009 at 10:41 AM, Richard Hirsch <hi...@gmail.com>wrote:
> Daniel thought it might be associated with actions but wasn't sure.
>
> There were less than 4 users in the db and just two users working
> concurrently. I checked the trace files and didn't see anything.
>
> I'll have to watch it more clsely to see if I see any patterns.
>
> D.
> On Mon, Jun 22, 2009 at 5:29 PM, David Pollak <
> feeder.of.the.bears@gmail.com
> > wrote:
>
> > What kind of user interaction is required to cause the problem or "just
> > running it"?
> >
> > On Sun, Jun 21, 2009 at 11:59 PM, Richard Hirsch <hirsch.dick@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > After the latest deployment on stax, we've been having a few stability
> > > problems where CPU and memory max out (see attached picture) with very
> > few
> > > users. A restart is necessary to be able to use the app again.
> > >
> > > Maybe, someone could check to see if we have some sort of stability
> > issues.
> > >
> > > Thanks.
> > >
> > > D.
> > >
> > >
> >
> >
> >
> > --
> > Lift, the simply functional web framework http://liftweb.net
> > Beginning Scala http://www.apress.com/book/view/1430219890
> > Follow me: http://twitter.com/dpp
> > Git some: http://github.com/dpp
> >
>
--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp
Re: Memory usage -> stability probs?
Posted by Richard Hirsch <hi...@gmail.com>.
Daniel thought it might be associated with actions but wasn't sure.
There were less than 4 users in the db and just two users working
concurrently. I checked the trace files and didn't see anything.
I'll have to watch it more clsely to see if I see any patterns.
D.
On Mon, Jun 22, 2009 at 5:29 PM, David Pollak <feeder.of.the.bears@gmail.com
> wrote:
> What kind of user interaction is required to cause the problem or "just
> running it"?
>
> On Sun, Jun 21, 2009 at 11:59 PM, Richard Hirsch <hirsch.dick@gmail.com
> >wrote:
>
> > Hi,
> >
> > After the latest deployment on stax, we've been having a few stability
> > problems where CPU and memory max out (see attached picture) with very
> few
> > users. A restart is necessary to be able to use the app again.
> >
> > Maybe, someone could check to see if we have some sort of stability
> issues.
> >
> > Thanks.
> >
> > D.
> >
> >
>
>
>
> --
> Lift, the simply functional web framework http://liftweb.net
> Beginning Scala http://www.apress.com/book/view/1430219890
> Follow me: http://twitter.com/dpp
> Git some: http://github.com/dpp
>
Re: Memory usage -> stability probs?
Posted by David Pollak <fe...@gmail.com>.
What kind of user interaction is required to cause the problem or "just
running it"?
On Sun, Jun 21, 2009 at 11:59 PM, Richard Hirsch <hi...@gmail.com>wrote:
> Hi,
>
> After the latest deployment on stax, we've been having a few stability
> problems where CPU and memory max out (see attached picture) with very few
> users. A restart is necessary to be able to use the app again.
>
> Maybe, someone could check to see if we have some sort of stability issues.
>
> Thanks.
>
> D.
>
>
--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp