Re: VM options to run Geronimo

[Maxim Berkultsev <ma...@gmail.com>]

Hi, Matt!

Sorry for delay with the response - I had figured out all the issues
you've mentioned. Please see my comments below...

2006/3/31, Matt Hogstrom wrote:
> Hi Maxim,
>
> We need a heap dump to chase this down.  We had a leak (identified in the
> release notes) when we shipped.  I found that it was about 1MB per hour under
> load on a dual 3.0Ghz Xeon (no hyperthreading).
>
> I was using Derby at the time.  I've been changing the tranQL connection manager
> and haven't seen it appear but I haven't specifically done a long run like I
> did before.
>
> It would be helpful to coordinate efforts on what areas your focusing on.

I have a question - do you think if it is reasonable and valuable up
to performance estimations to use Daytrader web primitives as workload
instances for JMeter?

> I think the memory leak would be excellent. Can you tell me a bit about your test
> setup?  I have a couple of engineering boxes you guys provided and they are
> outlined in the note you referred to earlier.

I suppose it is a memory leak, maybe the problem is with JMS - here is
the log I've got...

-----------------------
...
Geronimo Application Server started
12:37:34,127 INFO  [ActiveMQSession] Caught :java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
12:37:35,655 WARN  [ServletHandler] Error for /daytrader/servlet/PingServlet2MDB
Queue
java.lang.OutOfMemoryError
12:37:36,669 WARN  [BrokerClientImpl] caught exception consuming packet: PRODUCE
R_INFO: id = 4001 ProducerInfo{ clientId = 'ID:localhost-1990-1144139708721-10
:0' , destination = TradeBrokerQueue, producerId = '13905' , sessionId = '13905'
 , startTime = 1144139848263, started = false }
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError
12:37:39,663 WARN  [ServletHandler] Error for /daytrader/servlet/PingServlet2MDB
Queue
java.lang.OutOfMemoryError
12:37:42,673 WARN  [ServletHandler] Error for /daytrader/servlet/PingServlet2MDB
Queue
java.lang.OutOfMemoryError
12:37:43,640 WARN  [ServletHandler] Error for /daytrader/servlet/PingServlet2MDB
Queue
java.lang.OutOfMemoryError

-----------------------

I cannot provide a stack trace - but the problem is simply reproducible.

May be the problem is due to one cannot use PingServlet2MDBQueue for
multiuser load? Somehow I think it is a kind of memory leak. Since
Geronimo accepts the load for a minute or two and then fails with the
log...

>Also, can you do your testing with jRockit ?  I think the information
on another
>JVM would be a good thing.

jRockit 1.4.2 does work - no auxiliary VM memory options are
specified! I have not got any OOM problem while running the workload.

> Also, what TZ are you located in?  I'm UTC -5  (Raleigh, NC)

I'm GMT+3 (Moscow)

> Also, check us out on Irc irc://irc.freenode.net:6667 Channel #geronimo

Thank you.

--
Best regards,
Maxim Berkultsev, Intel Middleware Products Division

>
> Maxim Berkultsev wrote:
> > Hi, Matt!
> >
> > I've tried
> >
> > java -server -XX:-PrintTenuringDistribution -XX:+PrintGCDetails
> > -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -jar ./bin/server.jar
> >
> > since it seems too long to wait for the problem so I've limited VM with
> > the default heap size. Please see a portion of log below.
> >
> > It looks as if the heap was taken up intensively.
> >
> > For an OutOfMemory problem please see the stack trace further.
> >
> > Best regards,
> > Maxim Berkultsev, Intel Middleware Products Division
> >
> > ------------------
> >
> > .......
> >
> > 535.751: [Tenured[Unloading class
> > sun.reflect.GeneratedSerializationConstructorAccessor461]:
> > 58303K->58303K(58304K), 0.4943219 secs] 64355K->64332K(64832K), [Perm :
> > 31861K->31855K(32000K)] Heap after GC invocations=1056:
> >
> > Heap def new generation total 6528K, used 6028K [0x10010000, 0x10720000,
> > 0x10720000)
> >
> > eden space 5824K, 99% used [0x10010000, 0x105bffd8, 0x105c0000)
> >
> > from space 704K, 29% used [0x10670000, 0x106a3220, 0x10720000)
> >
> > to space 704K, 0% used [0x105c0000, 0x105c0000, 0x10670000)
> >
> > tenured generation total 58304K, used 58303K [0x10720000, 0x14010000,
> > 0x14010000)
> >
> > the space 58304K, 99% used [0x10720000, 0x1400fff8, 0x14010000, 0x14010000)
> >
> > compacting perm gen total 32000K, used 31855K [0x14010000, 0x15f50000,
> > 0x18010000)
> >
> > the space 32000K, 99% used [0x14010000, 0x15f2bdb8, 0x15f2be00, 0x15f50000)}
> > , 0.4963221 secs]
> >
> > 17:10:00,921 WARN [ServletHandler] Error for
> > /daytrader/servlet/PingServlet2MDBQueue java.lang.OutOfMemoryError
> >  ------------------
> >
> > 19:42:39,401 ERROR [Log] Error: PingServlet2MDBQueue.doGet(...):exception
> > posting message to TradeBrokerQueue destination
> >
> > 19:42:39,401 ERROR [Log] Error: PingServlet2MDBQueue.doGet(...): error
> >
> > java.util.ConcurrentModificationException
> > java.util.ConcurrentModificationException
> >
> > at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:448)
> >
> > at java.util.AbstractList$Itr.next(AbstractList.java:419)
> >
> > at java.util.AbstractCollection.remove(AbstractCollection.java:254)
> >
> > at org.activemq.TransactionContext.removeSession(TransactionContext.java
> > :116)
> >
> > at org.activemq.ra.RATransactionContext.removeSession(
> > RATransactionContext.java:57)
> >
> > at org.activemq.ActiveMQSession.doClose(ActiveMQSession.java:466)
> >
> > at org.activemq.ActiveMQSession.close(ActiveMQSession.java:447)
> >
> > at org.activemq.ra.JMSSessionProxy.cleanup(JMSSessionProxy.java:87)
> >
> > at org.activemq.ra.JMSSessionProxy.close(JMSSessionProxy.java:76)
> >
> > at org.apache.geronimo.samples.daytrader.web.prims.PingServlet2MDBQueue.
> >
> > doGet(PingServlet2MDBQueue.java:127)
> >
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
> >
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
> >
> > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:428)
> >
> > at org.apache.geronimo.jetty.JettyServletHolder.handle(
> > JettyServletHolder.java:99)
> >
> > at org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
> > WebApplicationHandler.java:830)
> >
> > at org.mortbay.jetty.servlet.JSR154Filter.doFilter(JSR154Filter.java:170)
> >
> > at org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
> > WebApplicationHandler.java:821)
> >
> > at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(
> > WebApplicationHandler.java:471)
> >
> > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:568)
> >
> > at org.mortbay.http.HttpContext.handle(HttpContext.java:1530)
> >
> > at org.mortbay.jetty.servlet.WebApplicationContext.handle(
> > WebApplicationContext.java:633)
> >
> > at org.mortbay.http.HttpContext.handle(HttpContext.java:1482)
> >
> > at org.mortbay.http.HttpServer.service(HttpServer.java:909)
> >
> > at org.mortbay.http.HttpConnection.service(HttpConnection.java:816)
> >
> > at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:982)
> >
> > at org.mortbay.http.HttpConnection.handle(HttpConnection.java:833)
> >
> > at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
> >
> > at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> >
> > at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> >
> > java.util.ConcurrentModificationException
> >
> > at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:448)
> >
> > at java.util.AbstractList$Itr.next(AbstractList.java:419)
> >
> > at java.util.AbstractCollection.remove(AbstractCollection.java:254)
> >
> > at org.activemq.TransactionContext.removeSession(TransactionContext.java
> > :116)
> >
> > at org.activemq.ra.RATransactionContext.removeSession(
> > RATransactionContext.java:57)
> >
> > at org.activemq.ActiveMQSession.doClose(ActiveMQSession.java:466)
> >
> > at org.activemq.ActiveMQSession.close(ActiveMQSession.java:447)
> >
> > at org.activemq.ra.JMSSessionProxy.cleanup(JMSSessionProxy.java:87)
> >
> > at org.activemq.ra.JMSSessionProxy.close(JMSSessionProxy.java:76)
> >
> > at org.apache.geronimo.samples.daytrader.web.prims.PingServlet2MDBQueue:
> >
> > doGet(PingServlet2MDBQueue.java:127)
> >
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
> >
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
> >
> > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:428)
> >
> > at org.apache.geronimo.jetty.JettyServletHolder.handle(
> > JettyServletHolder.java:99)
> >
> > at org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
> > WebApplicationHandler.java:830)
> >
> > at org.mortbay.jetty.servlet.JSR154Filter.doFilter(JSR154Filter.java:170)
> >
> > at org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
> > WebApplicationHandler.java:821)
> >
> > at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(
> > WebApplicationHandler.java:471)
> >
> > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:568)
> >
> > at org.mortbay.http.HttpContext.handle(HttpContext.java:1530)
> >
> > at org.mortbay.jetty.servlet.WebApplicationContext.handle(
> > WebApplicationContext.java:633)
> >
> > at org.mortbay.http.HttpContext.handle(HttpContext.java:1482)
> >
> > at org.mortbay.http.HttpServer.service(HttpServer.java:909)
> >
> > at org.mortbay.http.HttpConnection.service(HttpConnection.java:816)
> >
> > at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:982)
> >
> > at org.mortbay.http.HttpConnection.handle(HttpConnection.java:833)
> >
> > at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
> >
> > at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> >
> > at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> >
> > 19:43:29,178 WARN [ServletHandler] Error for
> > /daytrader/servlet/PingServlet2MDBQueue
> >
> > java.lang.OutOfMemoryError
> >
> > 19:43:30,365 WARN [HttpConnection] GET
> > /daytrader/servlet/PingServlet2MDBQueue HTTP/1.1
> >
> > java.lang.OutOfMemoryError
> >
> > 19:43:31,412 WARN [JournalMessageStore] Message could not be added to long
> > term store: null
> >
> > java.lang.OutOfMemoryError
> >
> > ----------
> >
> >
> > 2006/3/29, Matt Hogstrom wrote:
> >
> >>In the tests I'm running I use the following:
> >>
> >>java -server -Xmx2048m -Xms2048m -XX:-PrintTenuringDistribution
> >>-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -jar
> >>/home/hogstrom/geronimo-1.0/bin/server.jar
> >>
> >>I have not played too much with tuning the tenuring for the eden
> >>sizes.  Do you
> >>have a stack trace indicating where you failed?  OutOfMemory could mean
> >>several
> >>things.
> >>
> >>Maxim Berkultsev wrote:
> >>
> >>>Hi, all!
> >>>
> >>>I'm trying to make some performance evaluations of Geronimo with a help
> >>
> >>of
> >>
> >>>JMeter.
> >>>
> >>>It has appeared relatively simple to get Geronimo out of work. I've
> >>
> >>tried to
> >>
> >>>load it with JMeter and a web primitive called **PingServlet2MDBQueue**
> >>
> >>from
> >>
> >>>Daytrader bundle. I've created immediate load for 10 virtual users and
> >>>unlimited number of requests. Within a minute or two Geronimo stopped
> >>>responding to any request logging to console something like
> >>>...
> >>>18:32:56,180 WARN [ThreadedServer] EXCEPTION
> >>>java.lang.OutOfMemoryError
> >>>18:32:57,211 WARN [ThreadedServer] EXCEPTION
> >>>java.lang.OutOfMemoryError
> >>>...
> >>>
> >>>Has someone used any specific VM options to run Geronimo smoothly? (As
> >>
> >>for
> >>
> >>>me I've tried starting Geronimo with Java 1.4.2 Hotspot(TM) VM with
> >>
> >>-server
> >>
> >>>option enabled).
> >>>
> >>>Any advice or reference could be helpful. Thank you.
> >>>--
> >>>Best regards,
> >>>Maxim Berkultsev, Intel Middleware Products Division
> >>>
> >>
> >
>