RE: Tomcat 4.1 slower than Tomcat 3.3

[MOISSET Gérard OCISI <ge...@francetelecom.com>]

Solaris 8 TC 4.1.24 with Apache1.3.27 (EAPI), plug-in mod_jk-eapi.so with
"JVM 1.3.1_08"
----------------------------------------------------------------------------
------------

1)using "kernel tunable OS parameters" as following,
in /etc/sytem file,
* ajouts gestionnaire produits serveurs HTTP, web container 1/3 presentation
(increase sem..will decrease CPU load)
* file descritor limits
* soft
set rlim_fd_max=8192
* hard
set rlim_fd_cur=8192
* tcp_conn_hash_size
set tcp:tcp_conn_hash_size=8192
* shm,sem
set shmsys:shminfo_shmmax=4294967295
* set shmsys:shminfo_shmmax=1073741824
set shmsys:shminfo_shmmin=1
set shmsys:shminfo_shmmni=384
set shmsys:shminfo_shmmni=1000
set shmsys:shminfo_shmseg=64
set shmsys:shminfo_shmseg=200
* x2
set semsys:seminfo_semmni=20480
set semsys:seminfo_semmns=40960
set semsys:seminfo_semmnu=10220
set semsys:seminfo_semmsl=30048
set semsys:seminfo_semmap=256
set semsys:seminfo_semume=200

*Parametres reseau Ethernet a 100Mbps ou 1000Mbps Full Duplex (no autoneg..)
*pour toutes les interfaces.
*
set hme:hme_adv_autoneg_cap=0
set hme:hme_adv_100T4_cap=0
set hme:hme_adv_100fdx_cap=1
set hme:hme_adv_100hdx_cap=0
set hme:hme_adv_10fdx_cap=0
set hme:hme_adv_10hdx_cap=0

in /etc/rc2.d/S99net_config (see stack TCP),

echo "# Configuration parametres Reseau Platon ..."
# Pour inhiber le service ICMP de redirection du routage
ndd -set /dev/ip ip_ignore_redirect 1
ndd -set /dev/ip ip_send_redirects 0
# pour inhiber le service IP de redirection du routage
ndd -set /dev/ip ip_forwarding 0
ndd -set /dev/ip ip_strict_dst_multihoming 1
ndd -set /dev/ip ip_forward_directed_broadcasts 0
ndd -set /dev/tcp ip_enable_group_ifs 1
# Timers TCP
#ndd -set /dev/tcp tcp_time_wait_interval 60000
#ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 60000
# files TCP
ndd -set /dev/tcp tcp_conn_req_max_q0 4096
ndd -set /dev/tcp tcp_conn_req_max_q 1024
# Pour eviter la predictibilite du numero de sequence RFC1948
ndd -set /dev/tcp tcp_strong_iss 2
# inhiber le routage a la source de TCP
ndd -set /dev/tcp tcp_rev_src_routes 0
# Ajout gestionnaire produits serveur HTTP, web container 1/3 presentation
ndd -set /dev/tcp tcp_time_wait_interval 20000
ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 20000
ndd -set /dev/tcp tcp_keepalive_interval 10000
ndd -set /dev/tcp tcp_conn_req_max_q 60000
ndd -set /dev/tcp tcp_conn_req_max_q0 60000
ndd -set /dev/tcp tcp_xmit_hiwat 65536
ndd -set /dev/tcp tcp_recv_hiwat 65536
ndd -set /dev/tcp tcp_rexmit_interval_initial 500
ndd -set /dev/tcp tcp_rexmit_interval_min 1000
ndd -set /dev/tcp tcp_rexmit_interval_max 20000

3)tuning JVM parameters,

#echo "#####tuning TC#############################################"
CATALINA_HOME=/exec/commun/prog/tom4124
export CATALINA_HOME
CATALINA_PID="/exec/commun/site/tom4124/logs/tc.pid"
CATALINA_OUT=/exec/commun/site/tom4124
export CATALINA_OUT
echo "descripteurs de fichiers"
ulimit -n 8192
ulimit -a
echo "tuning de la JVM"
#mode exploitation, a redefinir en fonction du sizing de l'application si
besoin
#choix modele de threads  en mode One-to-one via Alternate libthread
uniquement (lwp) , specifique Solaris 8
LD_LIBRARY_PATH=/usr/lib/lwp:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
#choix modele de threads One-to-one via Alternate libthread (lwp),
specifique Solaris 8
#sur la JVM voir le modele de threads par defaut selon version 1.3 (Many to
Many) ou 1.4 (One to One)
JAVA_HOME=/exec/commun/prog/tom4124/j2sdk1_3_1_08
JAVA_OPTS="-hotspot -Xms128m -Xmx128m -XX:+UseBoundThreads -Xoptimize"
echo "version JAVA"
$JAVA_HOME/bin/java -version
echo "tuning JVM"
#mode profiling, a n'utiliser que sur analyse d'un point de contention,
jamais en exploitation
#JAVA_OPTS="-server -Xms256m -Xmx256m -runhprof:heap=all,cpu=times,format=a,
thread=y,doe=y,file=/exec/commun/site/tom4124/log
s/dump"
echo $JAVA_OPTS

4)tuning Apache process pool,

MinSpareServers 5
MaxSpareServers 64
StartServers 10
MaxClients 250
MaxRequestsPerChild 0
KeepAlive Off
Timeout 30

5)tuning TC Thread pool,

<Connector className="org.apache.ajp.tomcat4.Ajp13Connector" port="8009"
minProcessors="10" maxProcessors="350" enableLook
ups="false" connectionTimeout="30000" acceptCount="10" debug="0"
useURIValidationHack="false"/>
...

<Context path="/maquette" docBase="maquette" override="true"
privileged="true"  debug="0" reloadable="false"  >
<Logger className="org.apache.catalina.logger.FileLogger"
directory="/exec/commun/site/tom4124/logs"  prefix="maquette" suffix
=".log" timestamp="true" />
</Context>
....

you can change connectionTimeout="-1" if you have some trouble in tc log,
something like ERROR can't receive the response form Tomcat, network
problems,  err=0" in your log

6)Monitoring for effective results,
vmstat 5 (see r/q and idle cpu)
iostat -xpn 15 (see asvc_t)
prstat -a (see PS/NWLP.., no more than 600..)

7) Trouble shooting on bottleneck

using,

- "JVM profiling", because the bottleneck can be on your web application..

- "RICHP tool" because the bottleneck can be on tunable OS parameters or
tunable components parmaters with Solaris..

If people cant send me back a reply if everything can be improved a little
bit more.

Our first results seems to be ok with those parameters.

Solaris 8 TC 4.1.24 with Apache1.3.27 (EAPI), plug-in mod_jk-eapi.so with
"JVM 1.4.1_02" and JVM 1.3.1_08"
----------------------------------------------------------------------------
------------------------------

Now we can check if "One-to-one via Alternate libthread (lwp), specific for
Solaris 8"
and JVM with version 1.3 (Many to Many) ou 1.4 (One to One) can have the
same results with the same benchmark.

If every one can improve the tunable paramters or have already tested with
JVM 1.4 pls send me back a reply with your parameters


-----Message d'origine-----
De : Chris Agmen-Smith [mailto:chris.agmen-smith@Rightmove.co.uk]
Envoyé : lundi 7 avril 2003 14:51
À : 'tomcat-user@jakarta.apache.org'
Cc : 'rm@moosauer.de'
Objet : Re: Tomcat 4.1 slower than Tomcat 3.3


Reinhard,

Some good points. My comments in the text below:

> For me, it is faster, see below.
This seems to be the experience of JSP users. Velocity users, on the other
hand, are not quite so lucky, since recent optimizations seem to have
focussed on JSPs, not servlets.

> This is a hint not to look in the container but in your application.
As with all applications, ours could be tuned a little. An obvious start
point is link encoding (see below). However, the site runs very fast on
Tomcat 3.3 and poorly on 4.1, and we want to understand why, before we make
minor performance improvements in the code.

> My experience is:
> To compare the performance of a container (& JVM), its best to start
> with a minimal servlet or JSP. You can measure the > overhead for the
> whole request process.
I agree. Our minimal case is a simple Velocity template, with no database
hits. It still runs slower.

> I have used tomcat for 3 years now and I found the JVM has had more
> influence  to performance than the improvements of tc. (The old Sun
> Java2 JRE's had catastrophic performance, compared to IBMs JRE or
> the new 1.4 from Sun)
This sounds like a promising idea. Are there any alternative Java 1.4 JVMs
for Solaris? Linux and Windows seem to be replete with high-performance
alternatives, but what are the options for Solaris users?

> > We have tried profiling the app using hprof and these are the results:
> > Method                             Tomcat 3    Tomcat 4
> > java.net.URLEncoder.encode         33.62%      22.60%
> > java.net.SocketInputStream.read    14.15%      18.77%
> > java.net.SocketOutputStream.write   1.87%       3.47%
>
> You really should go to a more powerful profiler. This information is not
> enough to take action.
We used Optimizeit, and got the same results. It doesn't show us why Tomcat
4 chews up more CPU than Tomcat 3.

> My advice: Take your time and test thourougly, find an optimal solution.
I agree. Tomcat 3.3 is running very well indeed, we are under no time
pressure, we just want to find a servlet container that plugs in to JBoss
easily. I wonder if Tomcat 3.3 can run in-process with the latest
incarnation of JBoss? The main concern is that we want EJB calls to be
local, using EJB 2.0.

> Maybe it's a good time to check out where your application 'spends it's
time'.
> There could be a lot optimization possible, considering your long request
> times.
Yes - when we have found a servlet container that suits our application
better, then we will look at link encoding and Apache connectors to see if
we can't reduce CPU load and page generation time.

I would love to hear from other Velocity users out there who have had
similar experiences. Velocity is a great tool, and much easier to use than
JSP. Is there an environment or framework that is Velocity-compatible that
you would recommend? We use Turbine, but are looking for something newer and
more light-weight.

Thanks,

Chris Agmen-Smith


________________________________________________________________________
This email has been scanned for all viruses by the MessageLabs service.
________________________________________________________________________


---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org