You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Andrey Timofeyev <an...@gmail.com> on 2012/10/12 17:30:13 UTC
tomcat 7 APR Connector problems
Hi, everybody,
There is followen problem with tomcat 7.0.29 (With tomcat 6.0.18 there is
no such problem):
Any other services on the same machine lost connections with remote
services, when number of incoming connections to tomcat reach pollerSize.
(It seems that all file descriptors is used or something else)
As I see in the tomcat 7 latch was added in AprEndpoint, and if
connections reach maxLimit Acceptor locked on the latch.
Here is server configuration:
Linux 2.6.34.6-uni-02 #1 SMP Mon Sep 19 17:13:09 MSD 2011 x86_64 x86_64
x86_64 GNU/Linux
APR: libapr1-1.4.6-3.8
kernel.panic = 20
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
vm.min_free_kbytes = 65536
vm.swappiness = 0
net.core.optmem_max = 16777216
net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_max = 16777216
net.core.wmem_default = 16777216
net.ipv4.tcp_rmem = 4096 16777216 16777216
net.ipv4.tcp_wmem = 4096 16777216 16777216
net.core.netdev_max_backlog = 300000
net.core.somaxconn = 65536
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.tcp_max_tw_buckets = 1048576
net.ipv4.tcp_sack = 1
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
net.ipv4.neigh.default.gc_thresh1 = 2048
net.ipv4.neigh.default.gc_thresh2 = 4096
net.ipv4.neigh.default.gc_thresh3 = 8192
Is it tomcat issue or some misconfiguration?
--
Best regards,
Andrey.
Re: tomcat 7 APR Connector problems
Posted by Pid * <pi...@pidster.com>.
On 15 Oct 2012, at 07:42, Andrey Timofeyev <an...@gmail.com> wrote:
> Hi Chris,
>
> thank you so much for answer,
>
>> What about your Tomcat configuration:
> <Server port="8005" shutdown="SHUTDOWN">
> <Listener className="org.apache.catalina.core.AprLifecycleListener"
> SSLEngine="on"/>
You have no other listeners?
> <Service name="Catalina">
> <Connector
> maxPostSize="5242880"
> enableLookups="false"
> useBodyEncodingForURI="true"
> port="80"
> maxThreads="1100"
> disableUploadTimeout="true"
> maxHttpHeaderSize="8192"
> pollerSize="16768"
> maxKeepAliveRequests="-1"
> connectionTimeout="20000"
> asyncTimeout="110000"
> keepAliveTimeout="120000"
> compression="on"
>
> compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/x-javascript"
> address="{...}"
> acceptCount="1000"/>
> <Connector
> maxPostSize="5242880"
> enableLookups="false"
> useBodyEncodingForURI="true"
> port="82"
> maxThreads="100"
> disableUploadTimeout="true"
> maxHttpHeaderSize="8192"
> pollerSize="100"
> maxKeepAliveRequests="-1"
> connectionTimeout="20000"
> keepAliveTimeout="120"
> compression="on"
>
> compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/x-javascript"
> address="{...}"
> acceptCount="100"/>
> <Connector
> maxPostSize="5242880"
> enableLookups="false"
> useBodyEncodingForURI="true"
> port="443"
> maxThreads="1100"
> disableUploadTimeout="true"
> maxHttpHeaderSize="8192"
> pollerSize="32768"
> maxKeepAliveRequests="-1"
> connectionTimeout="20000"
> keepAliveTimeout="120000"
> compression="on"
>
> compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/x-javascript"
> address="0.0.0.0"
> acceptCount="100" scheme="https" secure="true"
> SSLEnabled="true"
> SSLCipherSuite="HIGH:MEDIUM:!ADH:!SSLv2"
> SSLCertificateFile="{...}"
> SSLCertificateKeyFile="{...}"
> SSLCertificateChainFile="{...}"/>
> <Engine name="Catalina" defaultHost="{...}">
> <Host name="{...}"
> appBase="{...}"
> autoDeploy="false"
> unpackWARs="true"
> deployXML="false">
> <Context path="" allowLinking="true" useHttpOnly="false">
> <Manager className="{customSessionManager}"
> maxActiveSessions="163840"
> maxInactiveInterval="1800"
> pathname=""/>
> </Context>
This Context seems to be missing a docBase, unless you are doing
something very bad and publishing the appBase.
Defining context in server.xml is not recommended.
p
> <Context path="/manager"
> privileged="true"
> docBase="{...}">
> <Realm
> className="org.apache.catalina.realm.MemoryRealm"
> pathname="conf/tomcat-users.xml"/>
> <Valve
> className="org.apache.catalina.valves.RemoteAddrValve"
> allow="{...}"/>
> </Context>
> <Context path="/share" docBase="{deploy.dir}/res/share"/>
> <Context path="/res" docBase="{deploy.dir}/res"/>
> <Context path="/web/gwt" docBase="{deploy.dir}/web/gwt"/>
> <Context path="/web/gwt-alt"
> docBase="{deploy.dir}/web/gwt-alt"/>
> </Host>
> </Engine>
> </Service>
> </Server>
>
>> Do you get exceptions or apparent deadlock? Can you generate a thread
> dump?
> There are no any deadlock or aniything strange compared to life server (i
> will generate it one yet time, may be i missed something)
>
>> Are your client connections long-lived
> Average client connection live 11 minutes.
>
>> What does 'ulimit -n' tell you
> 256k
>
> 2012/10/12 Christopher Schultz <ch...@christopherschultz.net>
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Andrey,
>>
>> On 10/12/12 11:30 AM, Andrey Timofeyev wrote:
>>> Hi, everybody,
>>>
>>> There is followen problem with tomcat 7.0.29 (With tomcat 6.0.18
>>> there is no such problem):
>>>
>>> Any other services on the same machine lost connections with
>>> remote services, when number of incoming connections to tomcat
>>> reach pollerSize. (It seems that all file descriptors is used or
>>> something else)
>>>
>>> As I see in the tomcat 7 latch was added in AprEndpoint, and if
>>> connections reach maxLimit Acceptor locked on the latch.
>>>
>>> Here is server configuration: Linux 2.6.34.6-uni-02 #1 SMP Mon Sep
>>> 19 17:13:09 MSD 2011 x86_64 x86_64 x86_64 GNU/Linux APR:
>>> libapr1-1.4.6-3.8 kernel.panic = 20
>>> net.ipv4.conf.all.accept_redirects = 0
>>> net.ipv4.conf.default.accept_redirects = 0
>>> net.ipv4.conf.all.rp_filter = 0 net.ipv4.conf.default.rp_filter =
>>> 0 net.ipv4.icmp_echo_ignore_broadcasts = 1 vm.min_free_kbytes =
>>> 65536 vm.swappiness = 0 net.core.optmem_max = 16777216
>>> net.core.rmem_max = 16777216 net.core.rmem_default = 16777216
>>> net.core.wmem_max = 16777216 net.core.wmem_default = 16777216
>>> net.ipv4.tcp_rmem = 4096 16777216 16777216 net.ipv4.tcp_wmem = 4096
>>> 16777216 16777216 net.core.netdev_max_backlog = 300000
>>> net.core.somaxconn = 65536 net.ipv4.tcp_max_orphans = 262144
>>> net.ipv4.tcp_max_syn_backlog = 65536 net.ipv4.tcp_max_tw_buckets =
>>> 1048576 net.ipv4.tcp_sack = 1 net.ipv4.tcp_synack_retries = 3
>>> net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_syn_retries = 3
>>> net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_window_scaling = 1
>>> net.ipv4.conf.all.arp_filter = 1 net.ipv4.conf.default.arp_filter =
>>> 1 net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.default.arp_ignore
>>> = 1 net.ipv4.conf.all.arp_announce = 2
>>> net.ipv4.conf.default.arp_announce = 2
>>> net.ipv4.neigh.default.gc_thresh1 = 2048
>>> net.ipv4.neigh.default.gc_thresh2 = 4096
>>> net.ipv4.neigh.default.gc_thresh3 = 8192
>>>
>>> Is it tomcat issue or some misconfiguration?
>>
>> What about your Tomcat configuration?
>>
>> Do you get exceptions or apparent deadlock? Can you generate a thread
>> dump? Are your client connections long-lived? How long? What does
>> 'ulimit -n' tell you?
>>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>>
>> iEYEARECAAYFAlB4PB4ACgkQ9CaO5/Lv0PB6owCfdKmTMEEiaZrlEOWwDSn8Zdic
>> P7MAn0dmd8U6FJMZvWEg89o8wWEuCqJV
>> =rdPD
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> --
> Best regards,
> Andrey.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 7 APR Connector problems
Posted by Andrey Timofeyev <an...@gmail.com>.
Hi Chris,
thank you so much for answer,
> What about your Tomcat configuration:
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on"/>
<Service name="Catalina">
<Connector
maxPostSize="5242880"
enableLookups="false"
useBodyEncodingForURI="true"
port="80"
maxThreads="1100"
disableUploadTimeout="true"
maxHttpHeaderSize="8192"
pollerSize="16768"
maxKeepAliveRequests="-1"
connectionTimeout="20000"
asyncTimeout="110000"
keepAliveTimeout="120000"
compression="on"
compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/x-javascript"
address="{...}"
acceptCount="1000"/>
<Connector
maxPostSize="5242880"
enableLookups="false"
useBodyEncodingForURI="true"
port="82"
maxThreads="100"
disableUploadTimeout="true"
maxHttpHeaderSize="8192"
pollerSize="100"
maxKeepAliveRequests="-1"
connectionTimeout="20000"
keepAliveTimeout="120"
compression="on"
compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/x-javascript"
address="{...}"
acceptCount="100"/>
<Connector
maxPostSize="5242880"
enableLookups="false"
useBodyEncodingForURI="true"
port="443"
maxThreads="1100"
disableUploadTimeout="true"
maxHttpHeaderSize="8192"
pollerSize="32768"
maxKeepAliveRequests="-1"
connectionTimeout="20000"
keepAliveTimeout="120000"
compression="on"
compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/x-javascript"
address="0.0.0.0"
acceptCount="100" scheme="https" secure="true"
SSLEnabled="true"
SSLCipherSuite="HIGH:MEDIUM:!ADH:!SSLv2"
SSLCertificateFile="{...}"
SSLCertificateKeyFile="{...}"
SSLCertificateChainFile="{...}"/>
<Engine name="Catalina" defaultHost="{...}">
<Host name="{...}"
appBase="{...}"
autoDeploy="false"
unpackWARs="true"
deployXML="false">
<Context path="" allowLinking="true" useHttpOnly="false">
<Manager className="{customSessionManager}"
maxActiveSessions="163840"
maxInactiveInterval="1800"
pathname=""/>
</Context>
<Context path="/manager"
privileged="true"
docBase="{...}">
<Realm
className="org.apache.catalina.realm.MemoryRealm"
pathname="conf/tomcat-users.xml"/>
<Valve
className="org.apache.catalina.valves.RemoteAddrValve"
allow="{...}"/>
</Context>
<Context path="/share" docBase="{deploy.dir}/res/share"/>
<Context path="/res" docBase="{deploy.dir}/res"/>
<Context path="/web/gwt" docBase="{deploy.dir}/web/gwt"/>
<Context path="/web/gwt-alt"
docBase="{deploy.dir}/web/gwt-alt"/>
</Host>
</Engine>
</Service>
</Server>
> Do you get exceptions or apparent deadlock? Can you generate a thread
dump?
There are no any deadlock or aniything strange compared to life server (i
will generate it one yet time, may be i missed something)
> Are your client connections long-lived
Average client connection live 11 minutes.
> What does 'ulimit -n' tell you
256k
2012/10/12 Christopher Schultz <ch...@christopherschultz.net>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Andrey,
>
> On 10/12/12 11:30 AM, Andrey Timofeyev wrote:
> > Hi, everybody,
> >
> > There is followen problem with tomcat 7.0.29 (With tomcat 6.0.18
> > there is no such problem):
> >
> > Any other services on the same machine lost connections with
> > remote services, when number of incoming connections to tomcat
> > reach pollerSize. (It seems that all file descriptors is used or
> > something else)
> >
> > As I see in the tomcat 7 latch was added in AprEndpoint, and if
> > connections reach maxLimit Acceptor locked on the latch.
> >
> > Here is server configuration: Linux 2.6.34.6-uni-02 #1 SMP Mon Sep
> > 19 17:13:09 MSD 2011 x86_64 x86_64 x86_64 GNU/Linux APR:
> > libapr1-1.4.6-3.8 kernel.panic = 20
> > net.ipv4.conf.all.accept_redirects = 0
> > net.ipv4.conf.default.accept_redirects = 0
> > net.ipv4.conf.all.rp_filter = 0 net.ipv4.conf.default.rp_filter =
> > 0 net.ipv4.icmp_echo_ignore_broadcasts = 1 vm.min_free_kbytes =
> > 65536 vm.swappiness = 0 net.core.optmem_max = 16777216
> > net.core.rmem_max = 16777216 net.core.rmem_default = 16777216
> > net.core.wmem_max = 16777216 net.core.wmem_default = 16777216
> > net.ipv4.tcp_rmem = 4096 16777216 16777216 net.ipv4.tcp_wmem = 4096
> > 16777216 16777216 net.core.netdev_max_backlog = 300000
> > net.core.somaxconn = 65536 net.ipv4.tcp_max_orphans = 262144
> > net.ipv4.tcp_max_syn_backlog = 65536 net.ipv4.tcp_max_tw_buckets =
> > 1048576 net.ipv4.tcp_sack = 1 net.ipv4.tcp_synack_retries = 3
> > net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_syn_retries = 3
> > net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_window_scaling = 1
> > net.ipv4.conf.all.arp_filter = 1 net.ipv4.conf.default.arp_filter =
> > 1 net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.default.arp_ignore
> > = 1 net.ipv4.conf.all.arp_announce = 2
> > net.ipv4.conf.default.arp_announce = 2
> > net.ipv4.neigh.default.gc_thresh1 = 2048
> > net.ipv4.neigh.default.gc_thresh2 = 4096
> > net.ipv4.neigh.default.gc_thresh3 = 8192
> >
> > Is it tomcat issue or some misconfiguration?
>
> What about your Tomcat configuration?
>
> Do you get exceptions or apparent deadlock? Can you generate a thread
> dump? Are your client connections long-lived? How long? What does
> 'ulimit -n' tell you?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iEYEARECAAYFAlB4PB4ACgkQ9CaO5/Lv0PB6owCfdKmTMEEiaZrlEOWwDSn8Zdic
> P7MAn0dmd8U6FJMZvWEg89o8wWEuCqJV
> =rdPD
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Best regards,
Andrey.
Re: tomcat 7 APR Connector problems
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Andrey,
On 10/12/12 11:30 AM, Andrey Timofeyev wrote:
> Hi, everybody,
>
> There is followen problem with tomcat 7.0.29 (With tomcat 6.0.18
> there is no such problem):
>
> Any other services on the same machine lost connections with
> remote services, when number of incoming connections to tomcat
> reach pollerSize. (It seems that all file descriptors is used or
> something else)
>
> As I see in the tomcat 7 latch was added in AprEndpoint, and if
> connections reach maxLimit Acceptor locked on the latch.
>
> Here is server configuration: Linux 2.6.34.6-uni-02 #1 SMP Mon Sep
> 19 17:13:09 MSD 2011 x86_64 x86_64 x86_64 GNU/Linux APR:
> libapr1-1.4.6-3.8 kernel.panic = 20
> net.ipv4.conf.all.accept_redirects = 0
> net.ipv4.conf.default.accept_redirects = 0
> net.ipv4.conf.all.rp_filter = 0 net.ipv4.conf.default.rp_filter =
> 0 net.ipv4.icmp_echo_ignore_broadcasts = 1 vm.min_free_kbytes =
> 65536 vm.swappiness = 0 net.core.optmem_max = 16777216
> net.core.rmem_max = 16777216 net.core.rmem_default = 16777216
> net.core.wmem_max = 16777216 net.core.wmem_default = 16777216
> net.ipv4.tcp_rmem = 4096 16777216 16777216 net.ipv4.tcp_wmem = 4096
> 16777216 16777216 net.core.netdev_max_backlog = 300000
> net.core.somaxconn = 65536 net.ipv4.tcp_max_orphans = 262144
> net.ipv4.tcp_max_syn_backlog = 65536 net.ipv4.tcp_max_tw_buckets =
> 1048576 net.ipv4.tcp_sack = 1 net.ipv4.tcp_synack_retries = 3
> net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_syn_retries = 3
> net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_window_scaling = 1
> net.ipv4.conf.all.arp_filter = 1 net.ipv4.conf.default.arp_filter =
> 1 net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.default.arp_ignore
> = 1 net.ipv4.conf.all.arp_announce = 2
> net.ipv4.conf.default.arp_announce = 2
> net.ipv4.neigh.default.gc_thresh1 = 2048
> net.ipv4.neigh.default.gc_thresh2 = 4096
> net.ipv4.neigh.default.gc_thresh3 = 8192
>
> Is it tomcat issue or some misconfiguration?
What about your Tomcat configuration?
Do you get exceptions or apparent deadlock? Can you generate a thread
dump? Are your client connections long-lived? How long? What does
'ulimit -n' tell you?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlB4PB4ACgkQ9CaO5/Lv0PB6owCfdKmTMEEiaZrlEOWwDSn8Zdic
P7MAn0dmd8U6FJMZvWEg89o8wWEuCqJV
=rdPD
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org