You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Sylvain Goulmy <sy...@gmail.com> on 2014/05/05 15:22:27 UTC
Re: Maximum number of JSP ?
Hi Christopher,
Thank you for your contribution to this thread. I think we we have made
good progress on the subject, here are some elements i'd like to share :
- The fact that the response time was increasing with the the number of JSP
loaded was linked to our monitoring tool... This tool hadn't the same
impact with websphere. Without monitoring the response time remains stable
no matter how many jsp are already loaded in the permgen.
- There is no permgen defined in the IBM JVM running Websphere and i was
wondering how much space the JVM was allocating to host this huge number of
JSP. The memory footprint of the process on the system was quite big : Xmx
: 1,5 Go, memory footprint of the JVM : 3,5 Go. It lets me think that
Websphere allocates a large space to host these JSP, i increase accordingly
the permgen size of my JVM to 1Go.
- I finally noticed that when the permgen is undersized (ie it cannot host
all the JSP of my application and has to unload class), the CPU impact is
much more important with the CMS garbage policy than with the parallel GC.
Our main concern so far was the CPU comsumption, we finally solved this by
tuning our monitoring tool correctly and by increasing the size of the
permgen.
Sylvain
On Mon, Apr 21, 2014 at 3:54 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Sylvain,
>
> This thread is bit old, but there were no conclusions si I figured I'd
> pick it back up.
>
> On 4/11/14, 11:52 AM, Sylvain Goulmy wrote:
> > Thank you all for your returns.
> >
> > Maybe also, just to give us an idea, you could tell us how much
> > memory that
> >> system has, and how much is given to use by Tomcat ?
> >
> >
> > Xmx is 1500 Mo. The PermSize is set so that it can easily load the
> > max number of JSP set with the maxLoadedJsp (set to 5000)
> > parameter.
>
> Assuming this is 1500MiB, then 16000 JSPs might have trouble fitting
> in here. Can you see how much space java.lang.Class objects are
> taking? How about how many JSPs are actually loaded? Presumably, you
> have 16000 .jsp files on the disk, but maybe only 1/2 of them are
> actually loaded before you have any problems.
>
> >> 1. Personally I am pessimistic about 'maxLoadedJsps' option,
> >> because even if you unload a JSP, the strings used in its code
> >> will still remain in JVM constant strings pool. Anyway, I expect
> >> the best performance would be if you do not try unloading the
> >> JSPs. 2. Did you set development=false on JspServlet?
>
> If Class objects are unloaded, they will not pin any data in the JVM
> string constant pool. You don't have anything to fear, here.
>
> What you *do* have to worry about is the buffers that JSP ends up
> using for custom tags. I can't remember the exact mechanics, but
> whenever a tag is executed, the content generated within it must be
> buffered in case of an error -- so an error message can be generated
> instead of failing "silently" to the remote user due to a
> response-already-committed problem.
>
> Anyway, when those buffers are expanded, they never shrink again. So,
> over time, all your buffers will grow enormous if you have a few
> places where this kind of thing can happen. The number of JSPs does
> not matter: you can do this with a single unfortunately-written JSP or
> with 16k of them. But given the extensive use of JSPs on your site,
> this is a likely candidate for any problems you are encountering.
>
> Are you getting OOMEs? Have you enabled verbose GC to see if the
> slowdowns are associated with GC pauses? It seems like you are coming
> to us with the problem "things are slow" and asking us to figure it
> out for you. Sorry, but you are going to need to provide more information.
>
> >>
> http://tomcat.apache.org/tomcat-7.0-doc/jasper-howto.html#Production_Configuration
> >>
> >>
> Note: I would not recommend using 'genStringAsCharArray',
> >> 3. There are a number of ways to monitor your memory usage. 4.
> >> Does the number of requests remain the same and only the number
> >> of JSPs increases? How much is "longer" in "the response time
> >> becomes longer" ?
> >
> >
> > 1. This is indeed a scenario i want to test, ie to size the PermGen
> > so it can load all the JSP without having to unload them. This
> > could require a huge PermGen size that i have to assess.
>
> java.lang.Class objects don't take a huge amount of memory. You will
> likely have to increase PermGen from the default, but you won't need a
> gigabyte or anything like that.
>
> > 2. development=false and i don't use the "genStringAsCharArray"
> > parameter
> >
> > 3. I'm using many tools which allows me to monitor the GC
> > behaviour
>
> What do those tools tell you?
>
> > 4. Here is the behaviour that i observed : i request the URL of a
> > jsp in a loop. The content of this JSP is always the same but it's
> > name is different in each URL so that it is considered as a new
> > one. This JSP is quite big (100 ko). Here is the evolution of the
> > response time :
> >
> > - First call : 70 ms - 1000th call : 100ms - 2000th call : 140 ms -
> > 3000th call : 200 ms
>
> The size of the .jsp file is not relevant. What is relevant is the
> "complexity". IF you just have 100kib of text, then there is no
> work/buffering to be done, for instance.
>
> > The more the permgen hosts many JSP, the more the response time
> > increases. On this test, the PermGen was big enough to host all the
> > JSP and the maxLoadedJsp parameter is set to 5000.
>
> What happens if you remove the maxLoadedJsp setting? Do you get an OOME?
>
> >> Is this performance problem something you didn't have before
> >> today?
> >
> > No indeed, we were on a different technology (Websphere).
>
> Same technology (JSP), different implementation. So, you can't give
> anyone some samples, eh? But you said you were able to replicate this
> with some simpler example? Did you use an example from your own web
> application or did you just create a sample .jsp and copy it to
> different names over and over again? If you help us reproduce the
> problem, we may be able to find a solution.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJTVSMsAAoJEBzwKT+lPKRYMS8P/2qOP8A74qPtRQL/1wXcw39p
> uJ1i9f3ml+vP2PB1Ujcxv2zeZIL8ztL/ClpJwk5nWh2mZW+S9ZUE15eozddzMn4d
> qbKT9brtTZWstX0HROY11yaMG52lQwsktw5TPwsLeRtSyvUSG2qTEohXYe8sRf/s
> ZMfYNdpgTED7RW+6iIooIN2YZMFSLUF6dyx0Up8sPRLFacDlGQw+v0BiLxyTSAev
> Fn1nsccGJ5wbSqVOZR8m8AtgCMzouAgxE7Ct0B/TI7d4ek7M1k9ba9mjk5xEvN/Y
> 4W/gJxr6f3NA0Mqw1zjhsIWjSUUf4o4q5ZhtoeihWWcBFaltO7dLymCFSxiXe/l3
> rM73PfFZTKSo8SIGi0sXPfCFqH+2hUMaABLLGhRI0UzFU65nO3mKYbLFaNNc+sej
> djNROjcmKt+MZs2JpUGttePjefeqhIlHWzgsvowb4QOob/ACm/r0xKN+byDR0NzP
> DQwLfxglnbxO95TwiuhEBVAKuMI4fVwNX0xNnoCQlh6K92R+nP9NZ4/PLWjkDKk/
> bB6HUJWJMw36i08xq1u+O/6bOYAeJZtt7fZFBlSz9ASGtpXDZOXhiao0JUOmrj9y
> Pd3zUFz1uY0+pymp3aamXTwSEI49OY3LfM1TIT09h2bqsZtIA+UYiMSLMTgY591/
> pwg8Ts0K4enWoVhq0P+5
> =1DhS
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Maximum number of JSP ?
Posted by Sylvain Goulmy <sy...@gmail.com>.
>
> Can you disclose what monitoring tool you use and explain how it
> impacted the measured value?
>
> Interesting. What monitoring tool are you using?
The monitoring tool used is Introscope from CA. You need to pay attention
to the "Deep Inheritance" parameter.
And, did it make a difference?
Yes, if the permgen is undersized, there is a significant impact on the CPU
consomption.
With "parallel GC" do you mean the default stop the world collector?
Indeed.
Did you also try G1 GC? I'd be curios to learn how well that did with
> your workload and especially how well it manages to keep GC times
> within given limit via -XX:MaxGCPauseMillis.
My application is running under JRE 1.6, i don't know if this mode is
available with this version.
The Garbage-First (G1) garbage collector is fully supported in Oracle JDK 7
> update 4 and later releases.
However i think that my application has not the required profile for this
GC mode :
Recommended Use Cases for G1
> The first focus of G1 is to provide a solution for users running
> applications that require large heaps with limited GC latency. This means
> heap sizes of around 6GB or larger, and stable and predictable pause time
> below 0.5 seconds.
I'm interested in what that tool was doing... and why it didn't seem
> to have the same effect under Websphere...
The agent version used is different in Tomcat and Websphere, but the
configuration is the same, especially the "Deep inhéritance" parameter.
Sylvain.
On Mon, May 5, 2014 at 5:02 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Sylvain,
>
> On 5/5/14, 9:22 AM, Sylvain Goulmy wrote:
> > Thank you for your contribution to this thread. I think we we have
> > made good progress on the subject, here are some elements i'd like
> > to share :
> >
> > - The fact that the response time was increasing with the the
> > number of JSP loaded was linked to our monitoring tool... This tool
> > hadn't the same impact with websphere. Without monitoring the
> > response time remains stable no matter how many jsp are already
> > loaded in the permgen.
>
> Interesting. What monitoring tool are you using?
>
> > - There is no permgen defined in the IBM JVM running Websphere and
> > i was wondering how much space the JVM was allocating to host this
> > huge number of JSP. The memory footprint of the process on the
> > system was quite big : Xmx : 1,5 Go, memory footprint of the JVM :
> > 3,5 Go. It lets me think that Websphere allocates a large space to
> > host these JSP, i increase accordingly the permgen size of my JVM
> > to 1Go.
>
> If you aren't specifically setting the JVM's heap and perm gen sizes,
> then you are getting the defaults. If you are using Websphere, then
> Websphere is likely setting those defaults for you, rather than just
> getting the JVMs defaults. I can't remember if IBM JVM even has a perm
> gen space like HotSpot-based (Sun/Oracle) JVMs.
>
> > - I finally noticed that when the permgen is undersized (ie it
> > cannot host all the JSP of my application and has to unload class),
> > the CPU impact is much more important with the CMS garbage policy
> > than with the parallel GC.
>
> Remember that you can configure your JVM to have different GC policies
> for each heap space (e.g. permgen versus rest).
>
> > Our main concern so far was the CPU comsumption, we finally solved
> > this by tuning our monitoring tool correctly and by increasing the
> > size of the permgen.
>
> I'm interested in what that tool was doing... and why it didn't seem
> to have the same effect under Websphere...
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJTZ6f8AAoJEBzwKT+lPKRYbxMP/R8+UJCkzEzYSZuwkT77PaIC
> KgUsh98UJylm2eUdmWqMAhBW0HiHgVz+tuhKPUVUnyOKaUCM2CvFx43sV8jjDfpF
> MiZP+ZdJFtbZ4/fW8zJ9Eg4v4bVJFfqDhIyfTU+wPGcRX/A88FQteyityYHdZqt7
> nqHtGDCi2UMD2CC2TucL815ITRcu1Jyrwse3SJjv5k5d0kLJXP3R4JZgtPRwYBit
> ptdUq9QDuQLV3AhYak5gax9WvupVsMJrJSjHa8aw0HUR7fqJlCKYe+6IJlTAmyv1
> x8Q5Ju3dCIigoaYcXfn0y+bpb0/DF4z9RoFiTA7p0mjViNkvYSL+RH7SYDJPbfDz
> Pe5QJPLdx+BxYPrq9XBAuTtgCRV2oefB5CnEMBUgm2tXqihMBmECS1ERYvo77vsh
> UUFkMUDannvsB6do4jQkIWr/i0RlSf40gHlDpnsTgmRO7kJM5fg9svBEMi2a3+Jo
> 4F7Ajjv7b8DYn4Dynh65P8VtwzFv3DKj14Mb5wUhXRkJFY0LpT4hUg999UGxEUq9
> cBQ+txfg3A72Rbn/ShkVefZGLWrsp0zqYzFGqhBlpV2HDYiJb2PpYcMX8B6gl4ez
> nUBwt6qMHDsx6+jH6ug2EGrFcTbVAM5aidN+CZquivE9Bts20rsgupWmFZWMPeaB
> wNsNG70ARwUcyeryJBC/
> =QUZb
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Maximum number of JSP ?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Sylvain,
On 5/5/14, 9:22 AM, Sylvain Goulmy wrote:
> Thank you for your contribution to this thread. I think we we have
> made good progress on the subject, here are some elements i'd like
> to share :
>
> - The fact that the response time was increasing with the the
> number of JSP loaded was linked to our monitoring tool... This tool
> hadn't the same impact with websphere. Without monitoring the
> response time remains stable no matter how many jsp are already
> loaded in the permgen.
Interesting. What monitoring tool are you using?
> - There is no permgen defined in the IBM JVM running Websphere and
> i was wondering how much space the JVM was allocating to host this
> huge number of JSP. The memory footprint of the process on the
> system was quite big : Xmx : 1,5 Go, memory footprint of the JVM :
> 3,5 Go. It lets me think that Websphere allocates a large space to
> host these JSP, i increase accordingly the permgen size of my JVM
> to 1Go.
If you aren't specifically setting the JVM's heap and perm gen sizes,
then you are getting the defaults. If you are using Websphere, then
Websphere is likely setting those defaults for you, rather than just
getting the JVMs defaults. I can't remember if IBM JVM even has a perm
gen space like HotSpot-based (Sun/Oracle) JVMs.
> - I finally noticed that when the permgen is undersized (ie it
> cannot host all the JSP of my application and has to unload class),
> the CPU impact is much more important with the CMS garbage policy
> than with the parallel GC.
Remember that you can configure your JVM to have different GC policies
for each heap space (e.g. permgen versus rest).
> Our main concern so far was the CPU comsumption, we finally solved
> this by tuning our monitoring tool correctly and by increasing the
> size of the permgen.
I'm interested in what that tool was doing... and why it didn't seem
to have the same effect under Websphere...
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTZ6f8AAoJEBzwKT+lPKRYbxMP/R8+UJCkzEzYSZuwkT77PaIC
KgUsh98UJylm2eUdmWqMAhBW0HiHgVz+tuhKPUVUnyOKaUCM2CvFx43sV8jjDfpF
MiZP+ZdJFtbZ4/fW8zJ9Eg4v4bVJFfqDhIyfTU+wPGcRX/A88FQteyityYHdZqt7
nqHtGDCi2UMD2CC2TucL815ITRcu1Jyrwse3SJjv5k5d0kLJXP3R4JZgtPRwYBit
ptdUq9QDuQLV3AhYak5gax9WvupVsMJrJSjHa8aw0HUR7fqJlCKYe+6IJlTAmyv1
x8Q5Ju3dCIigoaYcXfn0y+bpb0/DF4z9RoFiTA7p0mjViNkvYSL+RH7SYDJPbfDz
Pe5QJPLdx+BxYPrq9XBAuTtgCRV2oefB5CnEMBUgm2tXqihMBmECS1ERYvo77vsh
UUFkMUDannvsB6do4jQkIWr/i0RlSf40gHlDpnsTgmRO7kJM5fg9svBEMi2a3+Jo
4F7Ajjv7b8DYn4Dynh65P8VtwzFv3DKj14Mb5wUhXRkJFY0LpT4hUg999UGxEUq9
cBQ+txfg3A72Rbn/ShkVefZGLWrsp0zqYzFGqhBlpV2HDYiJb2PpYcMX8B6gl4ez
nUBwt6qMHDsx6+jH6ug2EGrFcTbVAM5aidN+CZquivE9Bts20rsgupWmFZWMPeaB
wNsNG70ARwUcyeryJBC/
=QUZb
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Maximum number of JSP ?
Posted by Robert Klemme <sh...@googlemail.com>.
Hi Sylvain,
thank you for sharing all these details!
On Mon, May 5, 2014 at 3:22 PM, Sylvain Goulmy <sy...@gmail.com> wrote:
> Hi Christopher,
>
> Thank you for your contribution to this thread. I think we we have made
> good progress on the subject, here are some elements i'd like to share :
> - The fact that the response time was increasing with the the number of JSP
> loaded was linked to our monitoring tool... This tool hadn't the same
> impact with websphere. Without monitoring the response time remains stable
> no matter how many jsp are already loaded in the permgen.
Can you disclose what monitoring tool you use and explain how it
impacted the measured value?
> - There is no permgen defined in the IBM JVM running Websphere and i was
> wondering how much space the JVM was allocating to host this huge number of
> JSP. The memory footprint of the process on the system was quite big : Xmx
> : 1,5 Go, memory footprint of the JVM : 3,5 Go. It lets me think that
> Websphere allocates a large space to host these JSP, i increase accordingly
> the permgen size of my JVM to 1Go.
And, did it make a difference?
> - I finally noticed that when the permgen is undersized (ie it cannot host
> all the JSP of my application and has to unload class), the CPU impact is
> much more important with the CMS garbage policy than with the parallel GC.
With "parallel GC" do you mean the default stop the world collector?
> Our main concern so far was the CPU comsumption, we finally solved this by
> tuning our monitoring tool correctly and by increasing the size of the
> permgen.
Did you also try G1 GC? I'd be curios to learn how well that did with
your workload and especially how well it manages to keep GC times
within given limit via -XX:MaxGCPauseMillis.
Kind regards
robert
--
[guy, jim].each {|him| remember.him do |as, often| as.you_can - without end}
http://blog.rubybestpractices.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org