You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by David kerber <dc...@verizon.net> on 2011/09/01 16:40:07 UTC
Re: Performance for many small requests
On 8/31/2011 12:25 PM, Tony Anecito wrote:
> Hi David,
>
> You need to not only look at the container but it's configuration and the jre that is being used. There have been a lot of improvements in all areas for performance. Also, understand the servlet model seems developers have completely forgotten about it and how important it is.
> Also, I always revaluate my design/implementation every 6 months or so and make changes based on lessons learned. Also, do not be afraid to try something new :]
I'm running JRE 6_27, in server mode, on windows server 2008 with 4
cores, 8GB RAM. TC 7.0.20, downloaded yesterday.
I'm having some somewhat minor performance issues, not performing quite
as well as my Win2k machine with TC 5.5. Could somebody look at my
server.xml and recommend some tweaks for handling tons of very small
requests, <150 bytes per request. The requests are sent with a single
http post, from ~600 remote sites collecting data every few seconds to
minutes.
Would one of the thread pools help this situation?
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener
className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<!--
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="300" minSpareThreads="4"/>
-->
<Connector port="1024"
protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"
maxThreads="600"
acceptCount="100"
minSpareThreads="10"
socketBuffer="16384"
/>
<!--
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="10000"
redirectPort="8443" />
-->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
</Host>
</Engine>
</Service>
</Server>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by Mark Thomas <ma...@apache.org>.
On 01/09/2011 15:40, David kerber wrote:
> On 8/31/2011 12:25 PM, Tony Anecito wrote:
>> Hi David,
>>
>> You need to not only look at the container but it's configuration and
>> the jre that is being used. There have been a lot of improvements in
>> all areas for performance. Also, understand the servlet model seems
>> developers have completely forgotten about it and how important it is.
>> Also, I always revaluate my design/implementation every 6 months or so
>> and make changes based on lessons learned. Also, do not be afraid to
>> try something new :]
>
> I'm running JRE 6_27, in server mode, on windows server 2008 with 4
> cores, 8GB RAM. TC 7.0.20, downloaded yesterday.
>
> I'm having some somewhat minor performance issues, not performing quite
> as well as my Win2k machine with TC 5.5. Could somebody look at my
> server.xml and recommend some tweaks for handling tons of very small
> requests, <150 bytes per request. The requests are sent with a single
> http post, from ~600 remote sites collecting data every few seconds to
> minutes.
I think the short answer is use a profiler. If your code is the problem,
fix it. If Tomcat is the problem, tell us where and we'll fix it.
> Would one of the thread pools help this situation?
Unlikely.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by "Darius D." <da...@gmail.com>.
Jeff Sturm wrote:
>
>
> The Linux JVM has a nice option -XX:+UseLargePages to help avoid TLB
> misses on heap accesses. It's good for modest gains, and has the side
> effect of locking the heap into memory. I tend to use it on systems with
> large heaps.
>
> Not much you can do about cache misses though, besides getting processors
> with more cache.
>
>
Yeah, Large(Huge) pages is big, free 10-15% perf if app is memory heavy. And
as a bonus you also reduce the size of kernel page translation tables (
works wonders if you are using for example Oracle that is using shared mem
for "instances", savings can be up to gigabytes on big systems ).
Even greater news is that recent Linux kernel versions ( i think 2.6.38 and
above ) have so called Transparent Huge Pages - that basically enable them
with some magic for all processes, giving 95% of HugePages benefits without
LargePages penalties (heap locking/pinning and app specific config). So
users of virtualisation, databases and JVMs with large heaps can rejoice :)
--
View this message in context: http://old.nabble.com/Performance-for-many-small-requests-tp32372424p32393854.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by Jeff Sturm <je...@eprize.com>.
> -----Original Message-----
> From: Darius D. [mailto:darius.ski@gmail.com]
> Sent: Saturday, September 03, 2011 1:36 PM
>
> As a side note - (CPU)cache/TLB misses have nothing to do with heap size.
> Too big heap size can be as bad as too low ( by stealing memory from OS that could
> have been used for file caches and other apps and increasing GC pauses ).
The Linux JVM has a nice option -XX:+UseLargePages to help avoid TLB misses on heap accesses. It's good for modest gains, and has the side effect of locking the heap into memory. I tend to use it on systems with large heaps.
Not much you can do about cache misses though, besides getting processors with more cache.
-Jeff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by "Darius D." <da...@gmail.com>.
David Kerber wrote:
>
> On 9/3/2011 1:15 PM, Darius D. wrote:
>>
>>
>> n828cl wrote:
>>>> From: Darius D. [mailto:darius.ski@gmail.com]
>>>> Subject: Re: Performance for many small requests
>>>> in my opinion using 64bit JVM with such a small
>>>> heap is only needed if performance testing shows
>>>> gains versus 32bit JVM.
>>> The main advantage of using a 64-bit JVM is the increased number of
>>> registers available in the x86-64 architecture, which can result in
>>> vastly
>>> reduced memory references. Whether or not that's important to overall
>>> performance is highly dependent on the application, of course.
>>>
>>>
>> Yeah, but I'd err on 32bit JVM side :) Gains from more registers are very
>> specific, but penalty from increased cache/TLB misses is big, and if you
>> start hitting hard page faults ( that would have been avoidable due to
>> lower
>> heap size with 32bit JVM ) - even one of those will erase all gains :)
>
> Then why shouldn't I just double my heap size? Wouldn't that eliminate
> the risk of increased cache misses? I was just using the settings from
> my 32-bit installation, but have plenty of RAM to allow me to increase
> the memory settings if that would help.
>
> Dave
>
Umm, sorry, it seems that 64 vs 32bit discussion has thrown this thread of
track. In your case it probably makes little difference. I'd definately
start by profiling and looking at gc logs.
As a side note - (CPU)cache/TLB misses have nothing to do with heap size.
Too big heap size can be as bad as too low ( by stealing memory from OS that
could have been used for file caches and other apps and increasing GC pauses
).
--
View this message in context: http://old.nabble.com/Performance-for-many-small-requests-tp32372424p32392937.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: David Kerber [mailto:dckerber@verizon.net]
> Subject: Re: Performance for many small requests
> Then why shouldn't I just double my heap size? Wouldn't
> that eliminate the risk of increased cache misses?
As Darius stated, this part of the discussion is probably completely irrelevant to any performance issues you have.
Regardless, doubling the heap size will likely _increase_ the cache misses, since you now have a larger target space being accessed through a fixed size cache space. This level of refinement is a real juggling act; unless your cores are staying very busy, it's unlikely to have any measurable effect.
You need to collect more data so you can start ruling out causes, and GC information is probably the easiest to start with.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by David Kerber <dc...@verizon.net>.
On 9/3/2011 1:15 PM, Darius D. wrote:
>
>
> n828cl wrote:
>>> From: Darius D. [mailto:darius.ski@gmail.com]
>>> Subject: Re: Performance for many small requests
>>> in my opinion using 64bit JVM with such a small
>>> heap is only needed if performance testing shows
>>> gains versus 32bit JVM.
>> The main advantage of using a 64-bit JVM is the increased number of
>> registers available in the x86-64 architecture, which can result in vastly
>> reduced memory references. Whether or not that's important to overall
>> performance is highly dependent on the application, of course.
>>
>>
> Yeah, but I'd err on 32bit JVM side :) Gains from more registers are very
> specific, but penalty from increased cache/TLB misses is big, and if you
> start hitting hard page faults ( that would have been avoidable due to lower
> heap size with 32bit JVM ) - even one of those will erase all gains :)
Then why shouldn't I just double my heap size? Wouldn't that eliminate
the risk of increased cache misses? I was just using the settings from
my 32-bit installation, but have plenty of RAM to allow me to increase
the memory settings if that would help.
Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by "Darius D." <da...@gmail.com>.
n828cl wrote:
>
>> From: Darius D. [mailto:darius.ski@gmail.com]
>> Subject: Re: Performance for many small requests
>
>> in my opinion using 64bit JVM with such a small
>> heap is only needed if performance testing shows
>> gains versus 32bit JVM.
>
> The main advantage of using a 64-bit JVM is the increased number of
> registers available in the x86-64 architecture, which can result in vastly
> reduced memory references. Whether or not that's important to overall
> performance is highly dependent on the application, of course.
>
>
Yeah, but I'd err on 32bit JVM side :) Gains from more registers are very
specific, but penalty from increased cache/TLB misses is big, and if you
start hitting hard page faults ( that would have been avoidable due to lower
heap size with 32bit JVM ) - even one of those will erase all gains :)
There are reasons why Linux and other stuff is shipped compiled optimized
for size, not with some fancy -O666 option.
--
View this message in context: http://old.nabble.com/Performance-for-many-small-requests-tp32372424p32392870.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Darius D. [mailto:darius.ski@gmail.com]
> Subject: Re: Performance for many small requests
> in my opinion using 64bit JVM with such a small
> heap is only needed if performance testing shows
> gains versus 32bit JVM.
The main advantage of using a 64-bit JVM is the increased number of registers available in the x86-64 architecture, which can result in vastly reduced memory references. Whether or not that's important to overall performance is highly dependent on the application, of course.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by "Darius D." <da...@gmail.com>.
Some good advice in this thread already, but given the power of server there
should be no problem serving even more requests ( as long as they are small,
not bound by CPU or I/O).
I'd start looking at JVM GC properties.
Turn on gc logging with
-Xloggc:/someplace/gclog_tomcat.txt -XX:+PrintGCDetails
( or some verbose / timestamped variant of this )
And see what your gc pauses look like, how frequent young gen / full gc are.
With hundreds of requests coming in every sec, you get into situation where
garbage collection pause will create a burst of requests once its done ( and
there is a risk that those burst objects will get promoted to old gen! ) and
cause stampede if server is near limits already. Generally You want your
requests to never hit old gen memory in JDK and get collected in some magic
parallel young gen GC collector ( -XX:+UseParNewGC, -XX:NewSize=???
-XX:MaxNewSize=??? -XX:SurvivorRatio=??? are worth checking / or even trying
G1C in JDK7 if You feel adventurous ).
As already was mentioned - finetuning JVM params is good idea, but only
after checking GC logs. Only thing i recommend outright is pinning JVM heap
with matching -Xmx and -Xms directives - or else JVM will keep
allocating/deallocating memory from OS, potentially creating fragmentation
issues in long term.
And there is a question of 64bit JVM, unless you need Java heap above ~1.5G,
32bit JVM should do just fine, otherwise you are just paying huge tax in
memory usage and CPU cache/TLB misses. -XX:+UseCompressedOops can help to
remove some of this tax, but in my opinion using 64bit JVM with such a small
heap is only needed if performance testing shows gains versus 32bit JVM.
--
View this message in context: http://old.nabble.com/Performance-for-many-small-requests-tp32372424p32392622.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chuck,
On 9/1/2011 4:52 PM, Caldarale, Charles R wrote:
>> From: Christopher Schultz [mailto:chris@christopherschultz.net]
>> Subject: Re: Performance for many small requests
>
>>> NIO may incur slightly more overhead due to thread switching.
>>> You'll have to measure to see if it's of any benefit.
>
>> Yes, but my guess is that it would be better than suffering
>> through keepalive timeouts if the clients stupidly specify
>> "Connection: Keep-Alive" when they don't actually intend to send
>> multiple requests.
>
> Agreed. I would forcibly disable keep-alive in the <Connector> by
> setting maxKeepAliveRequests to 1.
There's that, too :)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk5f8RwACgkQ9CaO5/Lv0PDnyACgrYfaaYrfukcgkgZHJNju/maT
2kQAoJFqXhy9jC0fqlZR9KAuYwCis9br
=TZML
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Subject: Re: Performance for many small requests
> > NIO may incur slightly more overhead due to thread switching.
> > You'll have to measure to see if it's of any benefit.
> Yes, but my guess is that it would be better than suffering through
> keepalive timeouts if the clients stupidly specify "Connection:
> Keep-Alive" when they don't actually intend to send multiple requests.
Agreed. I would forcibly disable keep-alive in the <Connector> by setting maxKeepAliveRequests to 1.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: Performance for many small requests
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chuck,
On 9/1/2011 2:00 PM, Caldarale, Charles R wrote:
>> From: David kerber [mailto:dckerber@verizon.net] Subject: Re:
>> Performance for many small requests
>
>>> You could also use the NIO connector which allows you to have
>>> fewer threads serve more requests without the
>>> keepalive-expiration delay.
>
>> Thanks, I'll take a look at this.
>
> NIO may incur slightly more overhead due to thread switching.
> You'll have to measure to see if it's of any benefit.
Yes, but my guess is that it would be better than suffering through
keepalive timeouts if the clients stupidly specify "Connection:
Keep-Alive" when they don't actually intend to send multiple requests.
>>> Looks like this connector has very little configuration. Is
>>> that because you aren't using it?
>
>> I didn't touch it because I don't know what to do with it, and
>> didn't know if deleting it would cause a problem.
>
> It's perfectly safe to comment it out or remove it. Doing so is
> unlikely to have any effect on performance.
It will probably result in the creation of at least one less thread.
It's also a simpler configuration, uses (slightly) less memory, and
reduces any attack vectors an intruder might want to use.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk5f7cwACgkQ9CaO5/Lv0PDyPwCeMKszXxJEetGetzqzrcSbFp4R
1uUAn1fM0ITBMDrgzjdHklbEmthx3cKi
=zvGH
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: David kerber [mailto:dckerber@verizon.net]
> Subject: Re: Performance for many small requests
> > You could also use the NIO connector which allows you
> > to have fewer threads serve more requests without the
> > keepalive-expiration delay.
> Thanks, I'll take a look at this.
NIO may incur slightly more overhead due to thread switching. You'll have to measure to see if it's of any benefit.
> > Looks like this connector has very little configuration.
> > Is that because you aren't using it?
> I didn't touch it because I don't know what to do with
> it, and didn't know if deleting it would cause a problem.
It's perfectly safe to comment it out or remove it. Doing so is unlikely to have any effect on performance.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by David kerber <dc...@verizon.net>.
On 9/1/2011 1:15 PM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> David,
>
> On 9/1/2011 10:40 AM, David kerber wrote:
>> I'm having some somewhat minor performance issues, not performing
>> quite as well as my Win2k machine with TC 5.5. Could somebody look
>> at my server.xml and recommend some tweaks for handling tons of
>> very small requests,<150 bytes per request. The requests are sent
>> with a single http post, from ~600 remote sites collecting data
>> every few seconds to minutes.
>
> If the requests are small and you are making them individually, you
> might want to either disable HTTP keepalives or have your clients
> specify "Connection: close" in their request headers. You could also
> use the NIO connector which allows you to have fewer threads serve
> more requests without the keepalive-expiration delay.
Thanks, I'll take a look at this.
>
>> Would one of the thread pools help this situation?
>
> Probably not, but I think thread pools (aka<Executor>s) are a good
> idea because they can take threads out-of-service when not in use.
>
>> <!--<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>> maxThreads="300" minSpareThreads="4"/> --> <Connector port="1024"
>> protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443"
>> maxThreads="600" acceptCount="100" minSpareThreads="10"
>> socketBuffer="16384" />
>>
>> <!--<Connector executor="tomcatThreadPool" port="8080"
>> protocol="HTTP/1.1" connectionTimeout="10000" redirectPort="8443"
>> /> -->
>>
>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>
> Looks like this connector has very little configuration. Is that
> because you aren't using it?
That's correct. I didn't touch it because I don't know what to do with
it, and didn't know if deleting it would cause a problem.
D
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
David,
On 9/1/2011 10:40 AM, David kerber wrote:
> I'm having some somewhat minor performance issues, not performing
> quite as well as my Win2k machine with TC 5.5. Could somebody look
> at my server.xml and recommend some tweaks for handling tons of
> very small requests, <150 bytes per request. The requests are sent
> with a single http post, from ~600 remote sites collecting data
> every few seconds to minutes.
If the requests are small and you are making them individually, you
might want to either disable HTTP keepalives or have your clients
specify "Connection: close" in their request headers. You could also
use the NIO connector which allows you to have fewer threads serve
more requests without the keepalive-expiration delay.
> Would one of the thread pools help this situation?
Probably not, but I think thread pools (aka <Executor>s) are a good
idea because they can take threads out-of-service when not in use.
> <!-- <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
> maxThreads="300" minSpareThreads="4"/> --> <Connector port="1024"
> protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443"
> maxThreads="600" acceptCount="100" minSpareThreads="10"
> socketBuffer="16384" />
>
> <!-- <Connector executor="tomcatThreadPool" port="8080"
> protocol="HTTP/1.1" connectionTimeout="10000" redirectPort="8443"
> /> -->
>
> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
Looks like this connector has very little configuration. Is that
because you aren't using it?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk5fvZcACgkQ9CaO5/Lv0PBoFgCgt/8pP7YiQsfn6QK2hQypuaSQ
XsgAn3znWPovPxKRfOmirkaJ1hPAVUG3
=rkks
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: David kerber [mailto:dckerber@verizon.net]
> Subject: Re: Performance for many small requests
> > For small requests having a large new generation in the heap helps.
> > This helps with the short term objects in the heap to get collected.
> I'm not sure what this means, but I'll do some digging.
In theory, current HotSpot GC algorithms should adjust the internal heap component sizes automatically, but in truly bizarre situations, one may benefit from manually controlling the boundaries. Approach any such adjustments with much caution. If your GC logging indicates that GC times are not a concern, I'd leave the finer details of heap management up to the JVM.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by David kerber <dc...@verizon.net>.
On 9/1/2011 12:09 PM, Tony Anecito wrote:
> Two things to think about in addition to the recommendation Chuck mentioned.
>
> 1. For small requests having a large new generation in the heap helps. This helps with the short term objects in the heap to get collected.
I'm not sure what this means, but I'll do some digging.
>
> 2. Lots of small disk IO may slow you down also. Remember http is synchronous and has to wait for disk IO unless it is spawned off to an independent thread.
I have write caching turned on in the OS, and went through the
synchronizing work in a previous round of performance tuning a couple of
years ago.
>
> Hope this helps.
> -Tony
>
> --- On Thu, 9/1/11, Caldarale, Charles R<Ch...@unisys.com> wrote:
>
>> From: Caldarale, Charles R<Ch...@unisys.com>
>> Subject: RE: Performance for many small requests
>> To: "Tomcat Users List"<us...@tomcat.apache.org>
>> Date: Thursday, September 1, 2011, 9:58 AM
>>> From: David kerber [mailto:dckerber@verizon.net]
>>
>>> Subject: Re: Performance for many small requests
>>
>>> Is there any indication from what I've said that
>>> I need a larger heap?
>>
>> Don't think so, but GC logging will tell you for
>> sure. The compressed OOPs capability with a small heap
>> should not incur any encode/decode overhead and should
>> improve cache hit rates.
>>
>> - Chuck
>>
>>
>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR
>> OTHERWISE PROPRIETARY MATERIAL and is thus for use only by
>> the intended recipient. If you received this in error,
>> please contact the sender and delete the e-mail and its
>> attachments from all computers.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by Tony Anecito <ad...@yahoo.com>.
Two things to think about in addition to the recommendation Chuck mentioned.
1. For small requests having a large new generation in the heap helps. This helps with the short term objects in the heap to get collected.
2. Lots of small disk IO may slow you down also. Remember http is synchronous and has to wait for disk IO unless it is spawned off to an independent thread.
Hope this helps.
-Tony
--- On Thu, 9/1/11, Caldarale, Charles R <Ch...@unisys.com> wrote:
> From: Caldarale, Charles R <Ch...@unisys.com>
> Subject: RE: Performance for many small requests
> To: "Tomcat Users List" <us...@tomcat.apache.org>
> Date: Thursday, September 1, 2011, 9:58 AM
> > From: David kerber [mailto:dckerber@verizon.net]
>
> > Subject: Re: Performance for many small requests
>
> > Is there any indication from what I've said that
> > I need a larger heap?
>
> Don't think so, but GC logging will tell you for
> sure. The compressed OOPs capability with a small heap
> should not incur any encode/decode overhead and should
> improve cache hit rates.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR
> OTHERWISE PROPRIETARY MATERIAL and is thus for use only by
> the intended recipient. If you received this in error,
> please contact the sender and delete the e-mail and its
> attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: David kerber [mailto:dckerber@verizon.net]
> Subject: Re: Performance for many small requests
> Is there any indication from what I've said that
> I need a larger heap?
Don't think so, but GC logging will tell you for sure. The compressed OOPs capability with a small heap should not incur any encode/decode overhead and should improve cache hit rates.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by David kerber <dc...@verizon.net>.
On 9/1/2011 11:36 AM, Caldarale, Charles R wrote:
>> From: David kerber [mailto:dckerber@verizon.net]
>> Subject: Re: Performance for many small requests
>
>>> Are you using 64-bit java or 32-bit?
>
>> 64-bit.
>
> Might want to try -XX:+UseCompressedOops, since you have a small heap on a 64-bit JVM.
I'll look that one up. Is there any indication from what I've said that
I need a larger heap? It doesn't look like it to me.
>
> - Chuck
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance for many small requests
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: David kerber [mailto:dckerber@verizon.net]
> Subject: Re: Performance for many small requests
> > Are you using 64-bit java or 32-bit?
> 64-bit.
Might want to try -XX:+UseCompressedOops, since you have a small heap on a 64-bit JVM.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by David kerber <dc...@verizon.net>.
On 9/1/2011 11:13 AM, Tony Anecito wrote:
> What is your current response time and what did you have before?
My issue isn't response time, it's number of requests per second handled.
> Are you using 64-bit java or 32-bit?
64-bit.
> What is your heap settings?
The "Initial memory pool" in tomcat7w is 256, and the max is 512. Task
manager is only showing 145MB allocated. Do those numbers refer to the
heap? If not, then I don't know where to look for that.
> Are you doing web services for these requests oris this straight html?
Straight html post, the app does some quick integrity checks and write
it to disk, then returns an "ok" response to the client.
>
> Regards,
> -Tony
>
> --- On Thu, 9/1/11, David kerber<dc...@verizon.net> wrote:
>
>> From: David kerber<dc...@verizon.net>
>> Subject: Re: Performance for many small requests
>> To: "Tomcat Users List"<us...@tomcat.apache.org>
>> Date: Thursday, September 1, 2011, 8:40 AM
>> On 8/31/2011 12:25 PM, Tony Anecito
>> wrote:
>>> Hi David,
>>>
>>> You need to not only look at the container but it's
>> configuration and the jre that is being used. There have
>> been a lot of improvements in all areas for performance.
>> Also, understand the servlet model seems developers have
>> completely forgotten about it and how important it is.
>>> Also, I always revaluate my design/implementation
>> every 6 months or so and make changes based on lessons
>> learned. Also, do not be afraid to try something new :]
>>
>> I'm running JRE 6_27, in server mode, on windows server
>> 2008 with 4 cores, 8GB RAM. TC 7.0.20, downloaded
>> yesterday.
>>
>> I'm having some somewhat minor performance issues, not
>> performing quite as well as my Win2k machine with TC
>> 5.5. Could somebody look at my server.xml and
>> recommend some tweaks for handling tons of very small
>> requests,<150 bytes per request. The requests are
>> sent with a single http post, from ~600 remote sites
>> collecting data every few seconds to minutes.
>>
>> Would one of the thread pools help this situation?
>>
>>
>> <Server port="8005" shutdown="SHUTDOWN">
>> <Listener
>> className="org.apache.catalina.core.AprLifecycleListener"
>> SSLEngine="on" />
>>
>> <Listener
>> className="org.apache.catalina.core.JasperListener" />
>>
>> <Listener
>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>> />
>>
>> <Listener
>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>> />
>>
>> <Listener
>> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>> />
>>
>> <GlobalNamingResources>
>> <Resource name="UserDatabase"
>> auth="Container"
>>
>> type="org.apache.catalina.UserDatabase"
>>
>> description="User database that can be updated and saved"
>>
>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>
>> pathname="conf/tomcat-users.xml" />
>> </GlobalNamingResources>
>>
>> <Service name="Catalina">
>>
>> <!--
>> <Executor name="tomcatThreadPool"
>> namePrefix="catalina-exec-"
>> maxThreads="300"
>> minSpareThreads="4"/>
>> -->
>> <Connector port="1024"
>> protocol="HTTP/1.1"
>> connectionTimeout="20000"
>> redirectPort="8443"
>> maxThreads="600"
>> acceptCount="100"
>> minSpareThreads="10"
>> socketBuffer="16384"
>> />
>>
>> <!--
>> <Connector executor="tomcatThreadPool"
>>
>> port="8080" protocol="HTTP/1.1"
>>
>> connectionTimeout="10000"
>>
>> redirectPort="8443" />
>> -->
>>
>> <Connector port="8009" protocol="AJP/1.3"
>> redirectPort="8443" />
>>
>> <Engine name="Catalina"
>> defaultHost="localhost">
>>
>>
>> <Realm
>> className="org.apache.catalina.realm.LockOutRealm">
>> <Realm
>> className="org.apache.catalina.realm.UserDatabaseRealm"
>>
>> resourceName="UserDatabase"/>
>> </Realm>
>>
>> <Host name="localhost"
>> appBase="webapps"
>> unpackWARs="true"
>> autoDeploy="true">
>>
>> </Host>
>> </Engine>
>> </Service>
>> </Server>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance for many small requests
Posted by Tony Anecito <ad...@yahoo.com>.
What is your current response time and what did you have before?
Are you using 64-bit java or 32-bit?
What is your heap settings?
Are you doing web services for these requests oris this straight html?
Regards,
-Tony
--- On Thu, 9/1/11, David kerber <dc...@verizon.net> wrote:
> From: David kerber <dc...@verizon.net>
> Subject: Re: Performance for many small requests
> To: "Tomcat Users List" <us...@tomcat.apache.org>
> Date: Thursday, September 1, 2011, 8:40 AM
> On 8/31/2011 12:25 PM, Tony Anecito
> wrote:
> > Hi David,
> >
> > You need to not only look at the container but it's
> configuration and the jre that is being used. There have
> been a lot of improvements in all areas for performance.
> Also, understand the servlet model seems developers have
> completely forgotten about it and how important it is.
> > Also, I always revaluate my design/implementation
> every 6 months or so and make changes based on lessons
> learned. Also, do not be afraid to try something new :]
>
> I'm running JRE 6_27, in server mode, on windows server
> 2008 with 4 cores, 8GB RAM. TC 7.0.20, downloaded
> yesterday.
>
> I'm having some somewhat minor performance issues, not
> performing quite as well as my Win2k machine with TC
> 5.5. Could somebody look at my server.xml and
> recommend some tweaks for handling tons of very small
> requests, <150 bytes per request. The requests are
> sent with a single http post, from ~600 remote sites
> collecting data every few seconds to minutes.
>
> Would one of the thread pools help this situation?
>
>
> <Server port="8005" shutdown="SHUTDOWN">
> <Listener
> className="org.apache.catalina.core.AprLifecycleListener"
> SSLEngine="on" />
>
> <Listener
> className="org.apache.catalina.core.JasperListener" />
>
> <Listener
> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
> />
>
> <Listener
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> />
>
> <Listener
> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
> />
>
> <GlobalNamingResources>
> <Resource name="UserDatabase"
> auth="Container"
>
> type="org.apache.catalina.UserDatabase"
>
> description="User database that can be updated and saved"
>
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>
> pathname="conf/tomcat-users.xml" />
> </GlobalNamingResources>
>
> <Service name="Catalina">
>
> <!--
> <Executor name="tomcatThreadPool"
> namePrefix="catalina-exec-"
> maxThreads="300"
> minSpareThreads="4"/>
> -->
> <Connector port="1024"
> protocol="HTTP/1.1"
> connectionTimeout="20000"
> redirectPort="8443"
> maxThreads="600"
> acceptCount="100"
> minSpareThreads="10"
> socketBuffer="16384"
> />
>
> <!--
> <Connector executor="tomcatThreadPool"
>
> port="8080" protocol="HTTP/1.1"
>
> connectionTimeout="10000"
>
> redirectPort="8443" />
> -->
>
> <Connector port="8009" protocol="AJP/1.3"
> redirectPort="8443" />
>
> <Engine name="Catalina"
> defaultHost="localhost">
>
>
> <Realm
> className="org.apache.catalina.realm.LockOutRealm">
> <Realm
> className="org.apache.catalina.realm.UserDatabaseRealm"
>
> resourceName="UserDatabase"/>
> </Realm>
>
> <Host name="localhost"
> appBase="webapps"
> unpackWARs="true"
> autoDeploy="true">
>
> </Host>
> </Engine>
> </Service>
> </Server>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org